🚀 Unleash the Power of AI with Confluent Kafka: An Expert's Guide to mcp-confluent 🚀

Harness the power of Large Language Models (LLMs) to revolutionize your interaction with Confluent Kafka and Confluent Cloud! mcp-confluent is a cutting-edge Model Context Protocol (MCP) server implementation that empowers AI assistants to seamlessly manage your Kafka infrastructure through natural language. Imagine effortlessly creating topics, managing connectors, and executing Flink SQL statements, all with simple, intuitive commands.

Interactive Demos: Witness the Magic

Goose CLI: Command Kafka with Natural Language

Claude Desktop: AI-Powered Confluent Management

Your Comprehensive Guide to mcp-confluent

This guide provides everything you need to get started, from initial setup to advanced development.

🛠️ User Guide: Master the Art of AI-Driven Kafka Management

Getting Started: Your First Steps to AI-Powered Kafka

1. Craft Your .env Configuration: Begin by creating a .env file in your project's root directory. Populate it with the necessary configuration details, drawing inspiration from the example structure provided below.

2. Configure Your Environment: Tailor the .env file with your Confluent Cloud environment settings. Refer to the Configuration section for detailed explanations of each variable.

3. Node.js Installation: Ensure Node.js is installed on your system. We highly recommend using NVM (Node Version Manager) for streamlined Node.js version management.

   nvm install 22
   nvm use 22

Configuration: Fine-Tuning Your AI-Kafka Connection

Create a .env file in the root directory of your project with the following configuration:

# .env file
BOOTSTRAP_SERVERS="pkc-v12gj.us-east4.gcp.confluent.cloud:9092"
KAFKA_API_KEY="..."
KAFKA_API_SECRET="..."
KAFKA_REST_ENDPOINT="https://pkc-v12gj.us-east4.gcp.confluent.cloud:443"
KAFKA_CLUSTER_ID=""
KAFKA_ENV_ID="env-..."
FLINK_ENV_ID="env-..."
FLINK_ORG_ID=""
FLINK_REST_ENDPOINT="https://flink.us-east4.gcp.confluent.cloud"
FLINK_ENV_NAME=""
FLINK_DATABASE_NAME=""
FLINK_API_KEY=""
FLINK_API_SECRET=""
FLINK_COMPUTE_POOL_ID="lfcp-..."
CONFLUENT_CLOUD_API_KEY=""
CONFLUENT_CLOUD_API_SECRET=""
CONFLUENT_CLOUD_REST_ENDPOINT="https://api.confluent.cloud"
SCHEMA_REGISTRY_API_KEY="..."
SCHEMA_REGISTRY_API_SECRET="..."
SCHEMA_REGISTRY_ENDPOINT="https://psrc-zv01y.northamerica-northeast2.gcp.confluent.cloud"

Environment Variables Reference: A Deep Dive

Usage: Unleashing the Power of AI-Driven Kafka

This MCP server seamlessly integrates with various MCP clients, including Claude Desktop and Goose CLI/Desktop. The specific configuration and interaction will vary depending on your chosen client.

1. Build the Foundation: Follow the instructions in the Developer Guide to build and run the server from source.
2. Configure Your MCP Client: Configure your client (e.g., Claude, Goose) to connect to the address where this server is running (likely localhost with a specific port).
3. Start the Magic: Launch your MCP client, which will automatically initiate an instance of this MCP server locally.
4. Interact with Confluent: Use the client's intuitive interface to interact with Confluent Cloud resources.

Configuring Claude Desktop: AI Assistant at Your Service

1. Access Claude Desktop Configuration:

   • Mac: ~/Library/Application Support/Claude/claude_desktop_config.json
   • Windows: %APPDATA%\Claude\claude_desktop_config.json

2. Modify Configuration File:

   • Open the config file in your preferred text editor.
   • Add or modify the configuration using one of the following methods:
   Option 1: Run from source

   {
     "mcpServers": {
       "confluent": {
         "command": "node",
         "args": [
           "/path/to/confluent-mcp-server/dist/index.js",
            "--env-file",
           "/path/to/confluent-mcp-server/.env",
         ]
       }
     }
   }

   Option 2: Run from npx

   {
     "mcpServers": {
       "confluent": {
         "command": "npx",
         "args": [
           "-y"
           "@confluentinc/mcp-confluent",
           "-e",
           "/path/to/confluent-mcp-server/.env"
         ]
       }
     }
   }

   Replace /path/to/confluent-mcp-server/ with the actual path where you've installed this MCP server.

3. Restart Claude Desktop: Close and reopen Claude Desktop for the changes to take effect.

Configuring Goose CLI: Command-Line Power Meets AI

1. Run the Configuration Command:

   goose configure

2. Follow the Interactive Prompts:

   • Select Add extension
   • Choose Command-line Extension
   • Enter mcp-confluent as the extension name
   • Choose one of the following configuration methods:
   Option 1: Run from source

   node /path/to/confluent-mcp-server/dist/index.js --env-file /path/to/confluent-mcp-server/.env

   Option 2: Run from npx

   npx -y @confluentinc/mcp-confluent -e /path/to/confluent-mcp-server/.env

   Replace /path/to/confluent-mcp-server/ with the actual path where you've installed this MCP server.

🧑‍💻 Developer Guide: Dive Deep into mcp-confluent

Project Structure: A Blueprint for Understanding

/
├── src/                 # Source code
│   ├── confluent/       # Code related to Confluent integration (API clients, etc.)
│   ├── tools/           # Tool implementations (each tool in a separate file)
│   ├── index.ts         # Main entry point for the server
│   └── ...              # Other server logic, utilities, etc.
├── dist/                # Compiled output (TypeScript -> JavaScript)
├── openapi.json         # OpenAPI specification for Confluent Cloud
├── .env                 # Environment variables (example - should be copied and filled)
├── README.md            # This file
└── package.json         # Node.js project metadata and dependencies

Building and Running: From Source to Execution

1. Install Dependencies:

   npm install

2. Development Mode (watch for changes):

   npm run dev

3. Production Build (one-time compilation):

   npm run build

4. Start the Server:

   npm run start

Testing: Ensuring Reliability and Performance

MCP Inspector: Your Debugging Companion

# make sure you've already built the project either in dev mode or by running npm run build
npx @modelcontextprotocol/inspector node  $PATH_TO_PROJECT/dist/index.js --env-file $PATH_TO_PROJECT/.env

Adding a New Tool: Expanding the AI-Kafka Ecosystem

1. Add a new enum to the enum class ToolName.
2. Add your new tool to the handlers map in the ToolFactory class.
3. Create a new file, exporting the class that extends BaseToolHandler.
   1. Implement the handle method of the base class.
   2. Implement the getToolConfig method of the base class.
4. Once satisfied, add it to the set of enabledTools in index.ts.

Generating Types: Maintaining Type Safety

# as of v7.5.2 there is a bug when using allOf w/ required https://github.com/openapi-ts/openapi-typescript/issues/1474. need --empty-objects-unknown flag to avoid it
npx openapi-typescript ./openapi.json -o ./src/confluent/openapi-schema.d.ts --empty-objects-unknown

Contributing: Join the Community

We welcome bug reports and feedback in the form of Github Issues. For guidelines on contributing please see CONTRIBUTING.md