<aside> 🏠 Comcast’s Xfinity Home

Automate All The Things

I was a engineer with a focus on stability, throughput and scale when my team and I first built the 1.0 engine open source - ‣ - 59 Forks, 337 stars - used by Amex even

Highlights:

Rules Engine

Pluggable Languages and Libraries

Circuit breakers

Timeouts

Replay on Failure

Optionally embed on “devices”

TLDR:

Processing events is the goal. Here's the the flow for an incoming event:

1. Find relevant rules.
2. For each rule, evaluate the rule's condition.
3. For each set of bindings that satisfies the rule's condition, execute the rule's action.

We launched to 22 million live customers. We created a front-end using what we thought was the latest tech, Polymer and built deploy-able templates users could instantiate with a sentence based structure to automate a “thing”.

There it is. Glorious. IoT was fresh and new. We built one of the first opensource engines that would power low-latency orchestration. End to end the system could preform actions in under 500ms, so when you opened your door and you wanted a light to turn on, like magic, it would.

Then we built into embedded devices such as the home tablet and routers.

We literally powered an entirely new business unit: https://archive.is/dOIuW

My very talented team really rounded me out into the Principle I am today. We teased apart the best ideas of Rulio, I studied how to build and launch new products and together we launched the first cloud services right as Amazon turned on EC2 for the first time.

</aside>

<aside> ❤️ The Team

I can’t go any further without mentioning these cool human beings. I am the mentor, advisor, counselor, forever learner and staff engineer that I am, in part because of them.

Derek McGowan had me playing around with Go 0 r57 and literally helped bring docker to life from nothing. Derek taught me a deep and wide breadth of linux from the ground up. The kernel, glibc, userspace, drivers, everything. James Stephens tooling. you’ve built the thing. now you need to build tools to really use it. Bill Hathaway Practicality. No hidden side effects. He taught me how to build something people will use, everyday. Chinar Dingankar together we hooked up the first AWS Direct connects into co-locations. BGP, Microtek, Fortigate, VPN, Failover. She is why I love my homelab today. Trevor Gattis why you should never write any code in the first place. Michael Lu test. meaure. test again. Amol Dharap allowed me to deeply explore the algorithm world I would later use to help build cloud services Tim Kim taught me how to build products which feel great to use Michael O’Dell showed how to solve human needs paired with technology they love Vivien Mo create environments where people feel safe and fulfilled to be their best selves Punyanjan Sen how to launch a build something they can’t ignore

This dream team, helped make, me.

</aside>

<aside> ⚡ Cloud Service for Developers

Rulio was pretty cool—it still is. Initially, Rulio offered a simpler yet powerful rule engine ideal for scenarios where decisions are based on changing data and context. However, as developers' needs evolved, it became clear that the future demanded a high-performance state machine and rule-based processing capabilities, perfect for complex event processing. We recognized that the Xfinity Home IoT engine had potential beyond managing devices—it could orchestrate complex automated processes across our operations. This required rethinking our approach to process management, scalability, and reliability. Through collaboration and iteration, we developed a robust solution that significantly enhanced our operational efficiency.

The Shift to Sheens:

Recognizing these demands, we took the foundational ideas from Rulio, sunset the project, and rebuilt it into an open-source engine we called

‣

This transition was driven by a vision to create an engine capable of meticulous process execution and flexibility.

Expanding Functionality with Low-Code Solutions:

To enhance user interaction, we integrated the best of the low-code community by incorporating Node-RED. By replacing the Node.js backend with our open-source engine, we achieved a powerful and versatile platform that found wild success.

To create a platform capable of addressing complex event processing and automation needs, we knew that enhanced orchestration was essential. Leveraging the node-based, flow-oriented approach of Node-RED, we integrated various components such as sources, sinks, functions-as-a-service (FaaS), and scheduling actions into cohesive, powerful flows.

At the heart of our platform is the robust orchestration framework I mentioned earlier, ‣ . This framework runs each node as a finite state machine (FSM) and coordinates the flow of data and actions through interconnected nodes, ensuring smooth and efficient operations.

User Experience: Our platform is designed to be user-friendly, catering to both beginners and power users:

• Drag and Drop Interface: Users can easily drag and drop nodes onto the palette, visually wiring them together to create complex automation flows. This intuitive interface simplifies the process of building automation workflows, making it accessible even to those with minimal programming experience.
• Wizard-Like Setup: For beginners, nodes come with wizard-like configurations that guide users through the setup process step-by-step. This helps users understand the functionality of each node and how to configure it for their specific needs.
• Composite Components: We augmented Node-Red, adding a powerful ability so users can create “composite logic” components without leaving the editing experience. This allows them to quickly encapsulate any logic, such as something domain-specific and reuse it as a new component in the palette. By doing so, users can build more complex and reusable flows, enhancing productivity and consistency across projects.
• Progressive Skill Development: As users become more comfortable with the platform, they can delve deeper into the capabilities of each node. Advanced users can write their own functions in popular programming languages, integrating custom logic and extending the platform’s functionality.
• Empowering Users: The platform is designed to bring users forward with technology instead of leaving them behind. It creates an environment where users can start with basic automation tasks and gradually level up their skills to handle more complex scenarios, eventually running their own functions and contributing to the automation ecosystem.
• Social Development: The flows and components created in Node-RED are stored using JSON which can be easily imported and exported for sharing with others.

</aside>

<aside> ☎️

Rooted in IoT: The project’s roots in the Internet of Things (IoT) are deep, as I helped build the Comcast Xfinity Home IoT engine. This experience was crucial in ensuring the product’s success, delivering customer-facing features and internal automation solutions for Comcast’s technology community.

Integration Strategy: Our integration strategy for enhanced orchestration focuses on the following key aspects:

• Modularity: Each node (source, processing, sink, scheduler) is designed to be modular. This allows for easy updates, replacements, or additions without disrupting the overall system.
• Scalability: The platform is built to scale horizontally. By leveraging cloud infrastructure, we ensure that the system can handle increasing loads seamlessly.
• Resilience: The orchestration framework is designed with fault tolerance in mind. Redundant paths and failover mechanisms are in place to ensure continuous operation even in the event of node failures.
• Security: Given the hostile environment of the network, we implemented robust security measures. This includes data encryption, secure communication channels, and stringent access controls to protect both data and operations.

The Cloud Transformation: Deploying on AWS as a containerized service, we leveraged a comprehensive suite of services:

• DynamoDB: Fast and flexible NoSQL storage solutions.
• Aurora RDS: Unmatched performance for relational database needs.
• Fargate and ECS: Simplified container orchestration.
• Route53, EC2, and Load Balancers: Ensured accessibility and resilience.
• Redis and Memcached: Efficient caching.
• CloudFormation: Infrastructure as code (IaC) for streamlined management.

Real-World Applications: Our enhanced orchestration capabilities, modeled after Node-RED’s flow-based approach and powered by the Sheens engine, have been pivotal in various real-world applications, particularly in IoT and automation:

• IoT Device Management: Coordinating data from numerous IoT devices, processing it in real-time, and triggering appropriate actions or alerts based on predefined rules.
• Automated Workflows: Implementing automated workflows in enterprise environments, such as processing incoming data, performing analytics, and updating systems or notifying stakeholders.
• Event-Driven Processing: Handling events from multiple sources, processing them through FaaS nodes, and routing the outcomes to the appropriate sinks for further action or storage.
• Adaptability: Embracing new paradigms in an evolving technical landscape.
• Collaboration: Achievements born from collective effort towards a shared goal.
• Vision: Belief in the potential of our innovations driving us forward.

Lessons Learned and Key Principles: Our journey, from a specific IoT application to a broad platform for automation and cloud functions, is a story of transformation. It highlights not just the evolution of technology but also the growth of a team willing to question, innovate, and expand the boundaries of what's possible. Key principles that guided us include:

• Adaptability: Embracing new paradigms in an evolving technical landscape.
• Collaboration: Achievements born from collective effort towards a shared goal.
• Vision: Belief in the potential of our innovations driving us forward. </aside>

<aside> ⚡ Production Ready Architecture

Both Sheens and Rulio are libraries. They need infrastructure legs. In fact they needed a few more services to be, robust. A distributed scheduler service needed to be built, an event injest and transform service, https://github.com/Comcast/eel and a few others. Rulio contained some I/O such as http requests. Sheens was the final boss. It would contain no I/O. Ideal, right?

TLDR: EC2 → ECS → Fargate for compute DyanmoDB to store states ELB → distribute incoming traffic MSK → partitioning work in order CloudFormation → deploy

With that came: S3, Route53, Direct Connect, IAM, Redis, Elasticache, Aurora, IAM

And a whole smattering of tooling to fill the gaps. Deploy templating with tagging Sprint Closure and Release notes Releasing cloud service versions commit hashes Jenkins and Concourse jobs Cloud CLI like aws cli but for the set of internal services

</aside>