import"github.com/go-coldbrew/data-builder"Package databuilder compiles a set of builder functions into an execution plan with automatic dependency resolution, then runs them sequentially or in parallel.
A builder is a plain Go function whose signature encodes its inputs and output as types:
func(ctx context.Context, in1 StructA, in2 StructB) (StructC, error)
Rules enforced by IsValidBuilder:
- The first parameter must be context.Context.
- All remaining parameters must be concrete struct values (no pointers, no variadics, no primitives).
- The function must return exactly two values: a concrete struct and an error.
- Two registered builders cannot produce the same output struct.
- A builder cannot take its own output type as input.
Types are identified by their fully qualified "pkgpath.TypeName", so the dependency graph is built entirely from ordinary Go type information.
- Build a DataBuilder with New.
- Register builder functions with [DataBuilder.AddBuilders].
- Call [DataBuilder.Compile] with zero-valued instances of the structs the caller will supply at runtime. Compile topologically sorts the builders into stages, returning a Plan.
- Run the plan with [Plan.Run] (sequential) or [Plan.RunParallel] (bounded worker pool). Both return a Result.
- Read typed outputs from the result with Result.Get or GetFromResult from inside a builder.
A compiled Plan is side-effect free and safe to reuse across goroutines. [Plan.Replace] can swap a builder for a compatible one without recompiling, as long as the replacement's inputs are a subset of the original's.
[Plan.RunParallel] runs all builders in the same stage of the DAG concurrently, bounded by a caller-supplied worker count. A panic or error from any builder is surfaced back to the caller; subsequent stages do not start. Use MaxPlanParallelism to size the worker pool to the widest stage.
Function-name (runtime.FuncForPC) and struct-name (reflect.Type) resolutions are cached in process-global sync.Maps. Keys are stable for the life of the program, so the caches never evict. Hot-path effects (benchstat, count=6):
- Result.Get: ~4x faster single-threaded, ~11x faster under parallel load, zero allocations on hit.
- AddBuilders (warm cache): ~40% faster, ~60% fewer allocations.
- Per-resolution hits: ~10-15 ns/op, zero allocations.
Benchmarks live in benchmarks_test.go; run `make bench` to measure on your hardware.
BuildGraph renders the compiled plan to a graphviz file in any format graphviz supports (png, svg, dot, ...). Graphviz must be installed on the system.
- Constants
- Variables
- func AddResultToCtx(ctx context.Context, r Result) context.Context
- func BuildGraph(executionPlan Plan, format, file string) error
- func GetFromResult(ctx context.Context, obj any) any
- func IsValidBuilder(builder any) error
- func MaxPlanParallelism(pl Plan) (uint, error)
- type DataBuilder
- type Plan
- type Result
SupportPackageIsVersion1 is a compile-time assertion constant. Downstream packages reference this to enforce version compatibility.
constSupportPackageIsVersion1=truevar (
// ErrInvalidBuilder is returned when the builder is not validErrInvalidBuilder=errors.New("the provided builder is invalid")
// ErrInvalidBuilderKind is returned when the builder is not a functionErrInvalidBuilderKind=errors.New("invalid builder, should only be a function")
// ErrInvalidBuilderNumInput is returned when the builder does not have 1 inputErrInvalidBuilderNumOutput=errors.New("invalid builder, should always return two values")
// ErrInvalidBuilderFirstOutput is returned when the builder does not return a struct as first outputErrInvalidBuilderFirstOutput=errors.New("invalid builder, first return type should be a struct")
// ErrInvalidBuilderSecondOutput is returned when the builder does not return an error as second outputErrInvalidBuilderSecondOutput=errors.New("invalid builder, second return type should be error")
// ErrInvalidBuilderMissingContext is returned when the builder does not have a context as first inputErrInvalidBuilderMissingContext=errors.New("invalid builder, missing context")
// ErrInvalidBuilderInput is returned when the builder does not have a struct as inputErrInvalidBuilderInput=errors.New("invalid builder, input should be a struct")
// ErrInvalidBuilderOutput is returned when the builder does not have a struct as outputErrMultipleBuilderSameOutput=errors.New("invalid, multiple builders CAN NOT produce the same output")
// ErrSameInputAsOutput is returned when the builder has the same input and outputErrSameInputAsOutput=errors.New("invalid builder, input and output should NOT be same")
// ErrCouldNotResolveDependency is returned when the builder can not be resolvedErrCouldNotResolveDependency=errors.New("dependency can not be resolved")
// ErrMultipleInitialData is returned when the initial data is provided twiceErrMultipleInitialData=errors.New("initial data provided twice")
// ErrInitialDataMissing is returned when the initial data is not providedErrInitialDataMissing=errors.New("need complile time defined initial data to run")
)ErrWTF is the error returned in case we find dependency resolution related errors, please report this
varErrWTF=errors.New("what a terrible failure: this is likely a bug in dependency resolution, please report this")funcAddResultToCtx(ctx context.Context, rResult) context.ContextAddResultToCtx adds the given result object to context
this function should ideally only be used in your tests and/or for debugging modification made to Result obj will NOT persist
funcBuildGraph(executionPlanPlan, format, filestring) errorBuildGraph helps understand the execution plan, it renders the plan in the given format please note we depend on graphviz, please ensure you have graphviz installed
funcGetFromResult(ctx context.Context, objany) anyGetFromResult allows builders to access data built by other builders
this function enables optional access to data, your code should not rely on values being present, if you have explicit dependency please add them to your function parameters
funcIsValidBuilder(builderany) errorIsValidBuilder checks if the given function is valid or not
funcMaxPlanParallelism(plPlan) (uint, error)MaxPlanParallelism return the maximum number of buildes that can be exsecuted parallely for a given plan
this number does not take into account if the builder are cpu intensive or netwrok intensive it may not be benificial to run builders at max parallelism if they are cpu intensive
DataBuilder is the interface for DataBuilder
typeDataBuilderinterface {
// AddBuilders adds the builders to the DataBuilder. The builders are added to the DataBuilderAddBuilders(fn...any) error// Compile compiles the builders and returns a plan that can be used to run the builders// The initial data is used to resolve the dependencies of the builders. The initial data should be a struct that contains the fields that are used as input for the builders when this Plan is executed.Compile(initialData...any) (Plan, error)
}Example
package main
import (
"context""fmt""strings"
)
// lets say we have some data being produced by a set of functions// but we need to define how their interaction should be and how their dependency// should be resolvedtypeAppRequeststruct {
FirstNamestringCityNamestringUpperCaseboolLowerCasebool
}
typeAppResponsestruct {
Msgstring
}
typeNameMsgstruct {
Msgstring
}
typeCityMsgstruct {
Msgstring
}
typeCaseMsgstruct {
Msgstring
}
// Lets try to build a sample builder with some dependency// Assuming we have an App that acts on the request// processes it in multiple steps and returns a Response// we can think of this process as a series of functions// NameMsgBuilder builds name salutation from our AppRequestfuncNameMsgBuilder(_ context.Context, reqAppRequest) (NameMsg, error) {
returnNameMsg{
Msg: fmt.Sprintf("Hello %s!", req.FirstName),
}, nil
}
// CityMsgBuilder builds city welcome msg from our AppRequestfuncCityMsgBuilder(_ context.Context, reqAppRequest) (CityMsg, error) {
returnCityMsg{
Msg: fmt.Sprintf("Welcome to %s", req.CityName),
}, nil
}
// CaseMsgBuilder handles the case transformation of the messagefuncCaseMsgBuilder(_ context.Context, nameNameMsg, cityCityMsg, reqAppRequest) (CaseMsg, error) {
msg:=fmt.Sprintf("%s\n%s", name.Msg, city.Msg)
ifreq.UpperCase {
msg=strings.ToUpper(msg)
} elseifreq.LowerCase {
msg=strings.ToLower(msg)
}
returnCaseMsg{
Msg: msg,
}, nil
}
// ResponseBuilder builds Application response from CaseMsgfuncResponseBuilder(_ context.Context, mCaseMsg) (AppResponse, error) {
returnAppResponse{
Msg: m.Msg,
}, nil
}
funcmain() {
// First we build an object of the builder interfaceb:=New()
// Then we add all the builders// its okay to call `AddBuilders` multiple timeserr:=b.AddBuilders(
NameMsgBuilder,
CityMsgBuilder,
CaseMsgBuilder,
)
fmt.Println(err==nil)
// lets ass all builderserr=b.AddBuilders(ResponseBuilder)
fmt.Println(err==nil)
// next we we compile this into a plan// the compilation ensures we have a resolved dependency graph_, err=b.Compile()
fmt.Println(err!=nil)
// Why did we get the error ?// if we look at our dependency graph, there is no builder that produces AppRequest// in order of dependency resolution to work we need to tell// the Compile method that we will provide it some initial Data// we can do that by passing empty structs// compiler just needs the type, values will come in laterep, err:=b.Compile(AppRequest{})
fmt.Println(err==nil)
// once the Compilation has finished, we get an execution plan// the execution plan once created can be cached and is side effect free// It can be executed across multiple go routines// lets run the Plan, remember to pass in the initial valueresult, err:=ep.Run(
context.Background(), // context is passed on the buildersAppRequest{
FirstName: "Ankur",
CityName: "Singapore",
LowerCase: true,
},
)
fmt.Println(err==nil)
// once the execution is done, we can read all the values from the resultresp:=AppResponse{}
resp=result.Get(resp).(AppResponse)
fmt.Println(resp.Msg)
}true
true
true
true
true
hello ankur!
welcome to singapore
funcNew() DataBuilderNew Creates a new DataBuilder
Plan is the interface that wraps execution of Plans created by DataBuilder.Compile method.
typePlaninterface {
// Replace replaces the builder function used in compile with a different function. The builder function should be the same as the one used in AddBuildersReplace(ctx context.Context, from, toany) error// Run runs the builders in the plan. The initial data is used to resolve the dependencies of the builders. The initial data should be a struct that contains the fields that are used as input for the builders when this Plan is executed.Run(ctx context.Context, initValues...any) (Result, error)
// RunParallel runs the builders in the plan in parallel. The initial data is used to resolve the dependencies of the builders. The initial data should be a struct that contains the fields that are used as input for the builders when this Plan is executed.RunParallel(ctx context.Context, countuint, initValues...any) (Result, error)
}Example
b:=New()
err:=b.AddBuilders(DBTestFunc, DBTestFunc4)
fmt.Println(err==nil)
ep, err:=b.Compile(TestStruct1{})
fmt.Println(err==nil)
_, err=ep.Run(context.Background(), TestStruct1{})
fmt.Println(err==nil)
err=ep.Replace(context.Background(), DBTestFunc, DBTestFunc5)
fmt.Println(err==nil)
_, err=ep.Run(context.Background(), TestStruct1{})
fmt.Println(err==nil)
// Output:// true// true// CALLED DBTestFunc// CALLED DBTestFunc4// true// true// CALLED DBTestFunc5// CALLED DBTestFunc4// truetrue
true
CALLED DBTestFunc
CALLED DBTestFunc4
true
true
CALLED DBTestFunc5
CALLED DBTestFunc4
true
Result is the result of the Plan.Run method
typeResultmap[string]anyfuncGetResultFromCtx(ctx context.Context) ResultGetResultFromCtx gives access to result object at this point in execution
this function should ideally only be used in your tests and/or for debugging modification made to Result obj may or may not persist
func (rResult) Get(objany) anyResult.Get returns the value of the struct from the result if the struct is not found in the result, nil is returned
Generated by gomarkdoc