Show HN: Pyproc – Call Python from Go Without CGO or Microservices

25 points by acc_10000 7 hours ago

Hi HN!I built *pyproc* to let Go services call Python like a local function — *no CGO and no separate microservice*. It runs a pool of Python worker processes and talks over *Unix Domain Sockets* on the same host/pod, so you get low overhead, process isolation, and parallelism beyond the GIL.

*Why this exists*

* Keep your Go service, reuse Python/NumPy/pandas/PyTorch/scikit-learn. * Avoid network hops, service discovery, and ops burden of a separate Python service.

*Quick try (\~5 minutes)*

Go (app):

``` go get github.com/YuminosukeSato/pyproc@latest ```

Python (worker):

``` pip install pyproc-worker ```

Minimal worker (Python):

``` from pyproc_worker import expose, run_worker @expose def predict(req): return {"result": req["value"] * 2} if __name__ == "__main__": run_worker() ```

Call from Go:

``` import ( "context" "fmt" "github.com/YuminosukeSato/pyproc/pkg/pyproc" ) func main() { pool, _ := pyproc.NewPool(pyproc.PoolOptions{ Config: pyproc.PoolConfig{Workers: 4, MaxInFlight: 10}, WorkerConfig: pyproc.WorkerConfig{SocketPath: "/tmp/pyproc.sock", PythonExec: "python3", WorkerScript: "worker.py"}, }, nil) _ = pool.Start(context.Background()) defer pool.Shutdown(context.Background()) var out map[string]any _ = pool.Call(context.Background(), "predict", map[string]any{"value": 42}, &out) fmt.Println(out["result"]) // 84 } ```

*Scope / limits*

* Same-host/pod only (UDS). Linux/macOS supported; Windows named pipes not yet. * Best for request/response payloads ≲ \~100 KB JSON; GPU orchestration and cross-host serving are out of scope.

*Benchmarks (indicative)*

* Local M1, simple JSON: \~*45µs p50* and *\~200k req/s* with 8 workers. Your numbers will vary.

*What’s included*

* Pure Go client (no CGO), Python worker lib, pool, health checks, graceful restarts, and examples.

*Docs & code*

* README, design/ops/security docs, pkg.go.dev: [https://github.com/YuminosukeSato/pyproc](https://github.com/YuminosukeSato/pyproc)

*License*

* Apache-2.0. Current release: v0.2.x.

*Feedback welcome*

* API ergonomics, failure modes under load, and priorities for codecs/transports (e.g., Arrow IPC, gRPC-over-UDS).

---

Source for details: project README and docs. ([github.com][1])

[1]: https://github.com/YuminosukeSato/pyproc "GitHub - YuminosukeSato/pyproc: Call Python from Go without CGO or microservices - Unix domain socket based IPC for ML inference and data processin"