All Data and AI Weekly #202 -- 11-Aug-2025

( AI, Data, NiFi, Iceberg, Polaris, Streamlit, Flink, Kafka, Python, Java, SQL, Unstructured Data )

https://bsky.app/profile/paasdev.bsky.social

NiFi + AI + AI Data Cloud + Iceberg.

https://www.reddit.com/r/DataEngineeringForAI/hot/

Monthly NYC and Youtube Events

https://lu.ma/PINSAI

Code and Open Source Projects

AWS New York Summit https://github.com/tspannhw/conferences/tree/main/2025/awsny

Hex + Snowflake Hackathon https://github.com/tspannhw/hackathons/tree/main/2025-07-15

Apache NiFi + AI Agents + Cortex AI + Snowflake AISQL

https://github.com/tspannhw/TrafficAI/tree/main/Agents

https://github.com/tspannhw/transit-ridership

https://github.com/tspannhw/conferences

https://github.com/tspannhw/hackathons/tree/main/2025-07-15

Articles

Here's a curated selection of recent articles covering key topics in the Snowflake ecosystem, including the latest in AI, data engineering, performance, and governance.

Cortex AI & Agents

• Announcing OpenAI GPT-5 on Snowflake Cortex AI

  • This article discusses the integration of OpenAI's powerful GPT-5 model within Snowflake Cortex AI, highlighting its capabilities for advanced language tasks directly within the Snowflake ecosystem.

• Building AI Agents for Data Science

  • Learn how to leverage the power of OpenAI models and Snowflake to build intelligent, autonomous AI agents specifically for data science workflows, enabling faster analysis and automation.

• Scaling Unstructured Data with Cortex LLMs

  • This post explores a method for efficiently processing large volumes of unstructured data by scaling batch inference using Snowflake Cortex LLMs, providing practical insights for handling complex data types.

• Chat with your Snowflake Data in Microsoft 365 Copilot

• A Strategic Guide for Data Leaders

• AI Observability & Trust with Cortex Enterprise

• Introducing Open-SWE: An Open-Source Asynchronous Coding Agent

Data Engineering & Integration

• How to set up OpenFlow SharePoint Connector

  • A step-by-step guide on configuring the OpenFlow SharePoint Connector to seamlessly integrate and move data between SharePoint and Snowflake.

• When your Kafka Sink Connector Writes Nothing

  • This debugging diary provides a detailed walkthrough of a common issue with the Snowflake Kafka Sink Connector, offering a practical solution and best practices.

• Real-Time Change Data Capture with OpenFlow

  • Discover how to implement real-time Change Data Capture (CDC) using Snowflake OpenFlow, ensuring your data warehouse is always up-to-date with the latest changes from source systems.

• OpenFlow Enabling World-Class FinOps

• From Apache Camel & NiFi to Snowflake Integration

• Simplify SQL Workflows with FLOW and UNION BY NAME

Performance & Cost Optimization

• New Search Optimization Features

  • An overview of the latest enhancements to Snowflake's search optimization service, designed to make data retrieval faster, more cost-effective, and easier to monitor.

• Cost Optimization in Snowflake: Best Practices

  • This article provides essential best practices for analytics engineers to optimize their Snowflake usage, helping to manage and reduce costs without sacrificing performance.

• Performance Tuning: A Strategic Guide

  • A comprehensive guide to strategically tuning your Snowflake environment for maximum performance, covering everything from query optimization to warehouse configuration.

• Data Governance & Security

• Simplify Sensitive Data Protection with Tag-Based Masking

  • Learn how to easily protect sensitive data in Snowflake using tag-based masking policies, ensuring compliance and data security with minimal effort.

• Governing Data in Snowflake: Tags, Policies, and Auditing

  • A deep dive into Snowflake's data governance features, focusing on how tags, policies, and auditing can be used to establish a robust and secure data environment.

Other Key Topics

• The Unofficial Snowflake Monthly Release Notes (July 2025)

  • A summary of the key features and updates from Snowflake's July 2025 release, providing a concise overview of the latest platform changes.

• 5 Expert-Level Snowflake Use Cases

  • Explore five advanced and innovative use cases for Snowflake that go beyond the standard documentation, showcasing the platform's full potential.

• How to get the right data for agentic AI

  • This post offers insights into preparing and curating the right data to effectively train and deploy agentic AI models on Snowflake.

• How to Use Custom JDBC Drivers

  • Using Custom JDBC Drivers with Snowflake OpenFlow (Apache NiFi).

• Snowflake JSON Support

• Long Range WiFi

• Small TTS

• DataVault

• Snowflake API + Postman

Platform Insights

• Anatomy of a Snowflake Query

  • This article offers a comprehensive deep dive into how the Snowflake query execution engine works, giving you a better understanding of performance and optimization.

• Why is Snowflake So Popular?

  • A strategic overview explaining the key factors behind Snowflake's rapid adoption and popularity in the modern data landscape.

• Faster, Cheaper, and More Transparent: New Search Optimization Features

  • An overview of the latest enhancements to Snowflake's search optimization service, designed to make data retrieval faster, more cost-effective, and easier to monitor.

Data Modeling & Engineering

• How We Built a Scalable Enterprise Data Model with Snowflake and dbt

  • This post shares a team's journey in building a robust data model using Snowflake and dbt, including valuable lessons learned and insights for future projects.

• Formatted SQL from Snowpark Python DataFrames

  • Learn how to generate clean, readable SQL code from your Snowpark Python dataframes, bridging the gap between Python-based data manipulation and traditional SQL workflows.

• Generate Reports at Scale using Snowpark Container Services (SPCS)

  • A guide on using Snowpark Container Services to generate and distribute reports efficiently, enabling you to scale your reporting tasks without infrastructure overhead.

AI & Natural Language

• Snowflake Intelligence Cheat Sheet

  • A quick reference guide to Snowflake Intelligence, providing a handy resource for understanding its key features and commands.

• The General-Purpose Snowflake MCP Server

  • This article introduces the concept of the MCP server, enabling users to execute SQL operations using natural language prompts, making data querying more accessible.

Case Studies

• Why Ventra Health Chose Snowflake

  • A case study on Ventra Health's decision to partner with Snowflake for its next-generation data and analytics platform, highlighting the benefits and outcomes of their implementation.

Documentation

This section contains links to official Snowflake documentation for key features.

• Snowflake Cortex AI SDK (Limited Access)

• Snowflake Cortex AI SQL

• Data Sharing Introduction

• Snowpipe Streaming High-Performance & Cost

• Snowpipe Streaming for Iceberg Tables

• OpenFlow Monitoring

• Flow Operators

• Externally Managed Iceberg Tables

Examples & Quickstarts

Check out these quickstarts and examples to see Snowflake in action and get hands-on with various use cases.

• E2E Task Graph Quickstart

• End-to-End Data Engineering with Snowpark Pandas

• Getting Started with your First Snowflake Notebook Project

• Building an AI Sales Assistant with Cortex

• Creating Semantic Models for Cortex Analyst

Code Repositories

Explore these GitHub repositories for code samples and open-source projects related to Snowflake.

• tspannhw/mta-snowpipe-streaming

• steipete/poltergeist

• Snowflake-Labs/sf-samples (ML Jobs and E2E Task Graph)

• Snowflake-Labs/semantic-model-generator

• lastmile-ai/mcp-agent

• IBM/mcp-context-forge

• Snowflake Intelligence Preview Guide

Videos

Watch these videos for visual tutorials and deep dives into key Snowflake features.

• OpenFlow with CrunchyBridge Postgresql

• Transit Data NiFi Snowflake

• NiFi + CortexAI

• MTA High Speed - Snowpipe Streaming 2

• RAG Chatbot

Events

Sign up for these upcoming virtual events to get hands-on experience and deep-dive into new features.

• Virtual Hands-on Lab: Build and Deploy Data Agents with Snowflake Cortex AI (August 21, 2025)

• Product Demo: Effortless Data Integration Ready for AI (OpenFlow) (August 13, 2025)

Sept 12 - Community over Code - Minneapolis - https://communityovercode.org/schedule/

Build Nov 4-6 - https://www.snowflake.com/en/build/

November 6, 2025 - NODES Conference - Virtual I will be speaking about Snowflake and Neo4J at their conference. https://neo4j.com/nodes-2025/

https://sessionize.com/tspann

https://github.com/timothyspann

© 2020-2025 Tim Spann https://www.youtube.com/@FLaNK-Stack