Staff Software Engineer

This is an exceptional opportunity to be a technical leader on the team that powers the heart of Prophecy: the Runtime, our orchestration and execution service. Every data workflow , whether generated by our AI data agents or built by a human in the visual editor , is planned, scheduled, and run at enterprise scale by Runtime, directly against the customer's own platform (Databricks, Snowflake, BigQuery, and more). When a business user asks an agent a question and sees trusted results seconds later, Runtime is what made it happen.

As a Staff Engineer, you will set the technical direction for this service: the distributed execution engine, the task coordinator, the multi-engine connectors, and the fast interactive previews that make the AI-native experience feel instant. You will own hard, ambiguous, high-leverage problems end-to-end , from architecture to production reliability , and raise the bar for the engineers around you.

* Own the engine behind the product. Architect and evolve the runtime service that turns agent- and human-authored data workflows into high-performance jobs that run reliably in production and interactively during authoring. * Make it fast and make it scale. Design the distributed execution and task-coordination layer , scheduling, concurrency, resource management, streaming of sample data (interims) , so that pipelines run efficiently from a single interactive preview to large production workloads. * Run everywhere the customer's data lives. Extend execution across multiple engines and warehouses , Databricks, Snowflake, BigQuery, Spark, dbt, Airflow , with first-class connectors, query pushdown, and consistent behavior across clouds. * Power the AI data agents. Build the execution path that lets our agents generate, run, validate, and refine pipelines in a tight loop , the difference between a plausible answer and a trusted one. * Own reliability and observability. Drive production robustness: fault tolerance, graceful degradation, recovery and restartability, monitoring, and the SLAs that enterprise customers depend on. * Lead technically. Set architecture and standards, drive cross-team designs, and mentor engineers. Own features and initiatives end-to-end, as a founder would , from the requirement to delivery.

* 5+ years of backend engineering experience building distributed systems or data-intensive services, with a track record of solving deep-tech problems * Deep expertise in distributed systems fundamentals: scheduling, task coordination, concurrency, resource management, and fault tolerance will help you succeed. * Strong proficiency in a systems language such as Go, Scala, or Java (our runtime is primarily Go and Python; functional-programming experience is a plus) * Hands-on experience running data at scale , with Spark and/or cloud data warehouses (Databricks, Snowflake, BigQuery) and query execution * A strong foundation in data structures, algorithms, and software design, with excellent analytical and debugging skills * Experience owning the reliability, performance, and observability of a production service * A demonstrated ability to lead technically , driving architecture, influencing across teams, and raising the quality bar for others * Comfort working in short, agile, iterative development cycles * Entrepreneurial spirit, priding yourself on getting things done with a high level of quality * Strong desire to work in a fast-moving startup environment with high ownership and attention to detail * You have been a top performer your entire career and hold yourself to very high standards * B Tech or above degree in CS or equivalent