Getting Started with AutoGenAI

AutoGenAI treats a generative AI workflow as a configuration containing a prompt, a provider, and a generation strategy. The package can benchmark these configurations using a task-specific scorer and then select configurations under quality, cost, and latency objectives.

Offline example

library(AutoGenAI)
ex <- autogenai_example()
fit <- optimize_ai(
  ex$task,
  ex$data,
  ex$providers,
  ex$prompts,
  temperatures = 0,
  strategies = "single"
)
fit
#> AutoGenAI optimization
#> Task: sentiment 
#> Configurations: 12 
#> 
#> Best configuration
#>  Provider: mock-balanced 
#>  Prompt ID: 3 
#>  Strategy: single 
#>  Temperature: 0 
#>  Quality: 1.0000 
#>  Cost: 0.000045 
#>  Latency: 0.0000 sec
#>  Utility: 0.9138

Pareto-efficient choices

pareto_ai(fit)
#>                         config_id prompt_id
#> 1  3::mock-balanced::0::single::1         3
#> 5  1::mock-balanced::0::single::1         1
#> 9     2::mock-cheap::0::single::1         2
#> 12    1::mock-cheap::0::single::1         1
#>                                                                                                                        prompt
#> 1  Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label. Check the answer carefully before responding.
#> 5                                                Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label.
#> 9                  Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label. Return only the final answer.
#> 12                                               Classify  he se ime  as posi ive,  ega ive, o   eu al. Re u  o ly  he label.
#>         provider temperature strategy samples   quality structured_score
#> 1  mock-balanced           0   single       1 1.0000000               NA
#> 5  mock-balanced           0   single       1 1.0000000               NA
#> 9     mock-cheap           0   single       1 0.6666667               NA
#> 12    mock-cheap           0   single       1 0.6666667               NA
#>    success_rate     cost      latency n   utility robustness
#> 1             1 4.50e-05 0.0000000000 6 0.9137905         NA
#> 5             1 3.35e-05 0.0001666667 6 0.8146759         NA
#> 9             1 8.20e-06 0.0000000000 6 0.3091236         NA
#> 12            1 6.70e-06 0.0001666667 6 0.1875000         NA

Robustness

st <- stress_test(
  ex$task,
  ex$data,
  ex$providers[[1]],
  ex$prompts[[1]]
)
st
#>   perturbation quality
#> 1     original       1
#> 2   whitespace       1
#> 3         case       1
#> 4         typo       1
#> 5   distractor       1
robustness_score(st)
#> [1] 1

Real providers

A real provider is any R function accepting prompt, input, and params and returning one text value. This deliberately keeps model-specific credentials and network behavior outside the package core.