Skip to content

Add JSON builder API to json_capi example; fix macOS-only .dylib assumptions - #1

Merged
yogthos merged 2 commits into
jolt-lang:mainfrom
burinc:feat/json-builder-api
Sep 14, 2026
Merged

yogthos merged 2 commits into
jolt-lang:mainfrom
burinc:feat/json-builder-api

Conversation

@burinc

@burinc burinc commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Closes #2

Summary

Two related additions, both born out of a real downstream use case (building an AWS Lambda custom runtime in Jolt — lambda-mvp-rst — that needed a way to build JSON, which Jolt itself has no library for):

  1. JSON builder API for examples/json/json_capi. The example previously only supported parsing/reading JSON (JsonValue::parse, field/array accessors). This adds the mirror-image write side: construct a serde_json::Value from Jolt and mutate it, entirely through Diplomat-generated FFI.
  2. Two cross-platform (macOS-only) portability fixes in bind.clj and runtime/src/diplomat/runtime.clj, found while loading this project's generated bindings on Linux (Amazon Linux 2023) for the first time — both previously assumed .dylib unconditionally.

1. JSON builder API

JsonValue gains:

  • Constructors: new_object, new_array, new_string, new_number, new_bool
  • Mutators: set_string, set_number, set_bool, set_value, push

Since these are &mut self methods, JsonValue also gains #[diplomat::opaque_mut] (previously only #[diplomat::opaque]), following the same pattern already used by the sdl3 example for mutable opaque types.

set_value/push clone the source serde_json::Value (mirroring the existing array_get accessor's ownership convention) — the caller's original handle remains independently valid and must still be closed separately.

demo.clj and leak-check.clj are both extended to exercise the new API. Leak-check results: 0 bytes leaked over 20,000 iterations using the builder API, with the existing negative-control case (deliberately not freeing a handle) still correctly flagged as a leak — confirming the leak detector itself wasn't disabled or bypassed by the new code path.

2. Portability fixes (.dylib → OS-detected)

Both bind.clj's shim-output filename and runtime/src/diplomat/runtime.clj's load! macro hardcoded the .dylib extension, so generated bindings could only ever be loaded on macOS. Fixed both via OS detection:

(if (str/includes? (str/lower-case (System/getProperty "os.name" "")) "mac") "dylib" "so")

This mirrors the pattern already in use in jolt-lang/http-client's tls.clj for OpenSSL library selection, so it's a proven approach for this codebase specifically (confirmed to work under Chez Scheme despite not running on a JVM).

Verification

  • cargo test --release in backend/: 11/11 pass, no regressions.
  • jolt run -m demo / jolt run -m leak-check in examples/json: still pass unchanged on macOS after the portability fixes (i.e. the fixes are additive, not behavior-changing, on the platform they previously assumed).
  • End-to-end downstream validation: this branch is vendored into lambda-mvp-rst (an AWS Lambda custom runtime written in Jolt), whose handler uses the new builder API to construct real JSON responses instead of hand-interpolating strings. That project's Docker/AL2023 build compiles this crate + runs bind.clj inside the container (exercising the Linux .so path these portability fixes enable) and successfully produces a working Lambda deployment — verified with a real aws lambda deploy / invoke / teardown cycle (cold init ~295–305 ms, warm invoke ~1.7–2.0 ms, consistent with this example's existing performance characteristics).

Compatibility

No breaking changes to the existing parse/read API surface — this is purely additive (new constructors/mutators) plus a portability fix that only changes behavior on non-macOS platforms (where bindings previously couldn't load at all).

…mptions

- examples/json/json_capi: add #[diplomat::opaque_mut] and builder
  constructors/mutators (new_object, new_array, new_string, new_number,
  new_bool, set_string, set_number, set_bool, set_value, push) to
  JsonValue, extending the existing parse/read-only API.
- Regenerate bindings/headers/shims (plain + mem-trace variants).
- Extend demo.clj and leak-check.clj to exercise and verify the new
  builder API (0 bytes leaked over 20,000 iterations; negative control
  confirms the leak detector itself still works).
- Fix two cross-platform bugs found while loading generated bindings on
  Linux (AL2023) for the first time, both previously hardcoded to macOS's
  .dylib extension:
  - runtime/src/diplomat/runtime.clj: load! macro now picks .dylib/.so
    via OS detection (mirrors the pattern already used in
    jolt-lang/http-client's tls.clj.
  - bind.clj: shim output filename uses the same OS detection.
- Verified: cargo test --release (11/11, backend, no regressions),
  jolt run -m demo / -m leak-check still pass unchanged on macOS.
EOF
)
Addresses code review on the JSON builder API PR:

- DiplomatStringView.len was measured with (count s) (a char count)
  instead of a UTF-8 byte count, silently truncating/corrupting any
  non-ASCII &str param. Now (alength (.getBytes s)).
- A plain (non-Option) bool return was emitted unwrapped; 0 is truthy
  in Clojure/Jolt, so callers checking the return value couldn't see
  failure. Now wrapped with (not= 0 ...), in both the shimmed and the
  direct-defcfn emission paths. Fixes json's new set_string/set_number/
  set_bool/set_value/push, and pre-existing semver::is_prerelease,
  semver::VersionReq::matches, and regex::is_match.
- load!'s syntax-quote leaked a reference to the private
  native-lib-ext var into every call site's namespace; now unquoted
  so the literal value is spliced in at macro-expansion time.
- Documented NaN/Infinity -> null substitution on new_number/set_number.
- Fixed a stale repo-name reference in json_capi's lib.rs comment.
- Added 4 backend unit tests covering both bugs above and the two new
  codegen shapes (no-arg ctor returning an owned opaque; &mut self
  with two &str params).

Regenerated generated/ and generated-traced/ for every affected
example crate (base64, chrono, json, markdown, regex, sdl3, semver,
tantivy). callback needed no change (no string params, no bool
returns). Rebuilt and re-ran the demo + leak-check for every crate
except sdl3, which fails to link locally on a missing SDL3_ttf system
library unrelated to this change; sdl3's regenerated bindings are
diff-verified against the fixed generator but not execution-verified.
@burinc
burinc force-pushed the feat/json-builder-api branch from 3595074 to 61e1e08 Compare September 13, 2026 02:29
@burinc

burinc commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a fix for the review feedback in 61e1e08.

#1 (byte-length, HIGH). DiplomatStringView.len was (count s), a char count, not a UTF-8 byte count. Fixed to (alength (.getBytes s)) in the generator. Re-ran the exact repros: "héllo" and the emoji string now round-trip intact, parse("{\"name\":\"José\"}") now parses.

#2 (truthy-0, HIGH). 0 is truthy in Clojure/Jolt, so an unwrapped bool return was unusable in (if ...). Fixed with (not= 0 ...) in both the shimmed and the direct-defcfn codegen paths (needed both, not just one, to fully cover it). This turned out to be pre-existing beyond json's new methods: semver::is_prerelease, semver::VersionReq::matches, and regex::is_match had the same bug, now fixed too.

#3. Left as bool-only, per your note that fixing #2 makes false observable. No throw added.

#4. Documented the NaN/Infinity -> null substitution on new_number/set_number. No behavior change.

#5 (private-var leak). load!'s syntax-quote now unquotes native-lib-ext so the literal value is spliced in at macro-expansion time, instead of leaking a reference to the private var into every call site's namespace.

#6 (test coverage). Added 4 backend unit tests: byte-length, both bool-wrap paths, the no-arg-ctor-returning-owned-opaque shape, and the &mut self two-string-param shape.

#7. Fixed the stale repo-name comment.

Regenerated generated/ and generated-traced/ for every affected example crate (base64, chrono, json, markdown, regex, semver, tantivy, url). callback needed no change. Rebuilt and re-ran the demo + leak-check for all of them except sdl3, which fails to link locally on a missing SDL3_ttf system library, unrelated to this change -- its regenerated bindings are diff-verified against the fixed generator but not execution-verified.

@burinc

burinc commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

@fricze Would love your view on the above. Thanks

@yogthos
yogthos merged commit 28bfc7f into jolt-lang:main Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

json_capi example: no way to build JSON, only parse it (+ .dylib-only bindings on Linux)

3 participants

⌂