awslabs/benchmark-ai

[Improvement] Increase error state visibility to end users

Open

#996 opened on Jan 7, 2020

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (6 forks)auto 404
enhancementgood first issue

Repository metrics

Stars
 (19 stars)
PR merge metrics
 (PR metrics pending)

Description

Currently, it's not necessarily clear to the user if an error has occured along the way. We'll use this ticket to track how we can increase this visibility.

Note: Status messages should be emitted to this status topic.

There are a few different sources for errors, including, but not limited to:

  • Toml validation errors (bff)
  • Problems fetching data (fetcher)
  • Problems transpiling or creating kubernetes resources (executor)
  • Problems with the execution of the benchmark in k8s (should be caught by the watcher)
  • Problems executing a benchmark on SageMaker (sm-executor)

On the python side, most (if not all) services extend the the KafkaService class, which contains utility method that can be used to emit status messages.

For the bff, there is a function that can be used to emit status events.

My suggestion is to go through each of the services and ensure that whichever method is handling an event appropriately catches any errors and emits a helpful error status message.

Contributor guide

Research direction
Work through each service listed (bff, fetcher, executor, watcher, sm executor) and add error handling that emits status messages to the status topic. Use the provided utility methods in KafkaService and bff events to emit error status messages.
Tech stack
pythonclojurekafkakubernetesaws
Domain
backendinfrastructure
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
PythonClojureKafkaKubernetes
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30