opensearch-project/alerting-dashboards-plugin

[BUG] Tracking issue for known flaky tests

Open

#415 opened on Dec 9, 2022

View on GitHub
 (8 comments) (0 reactions) (0 assignees)JavaScript (118 forks)auto 404
buggood first issue

Repository metrics

Stars
 (31 stars)
PR merge metrics
 (PR metrics pending)

Description

Please add more flaky tests to this issue as they're discovered; or notes from investigations into the causes of the flaky tests. Reducing flakiness will increase confidence in our automated testing, and reduce the amount of time needed to merge changes as rerunning cypress tests is a time-consuming process.

List of known flaky tests

  1. bucket_level_monitor_spec.js > Bucket-Level Monitors > can be created > by visual editor
    1. This test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails.
CypressError: Timed out retrying after 10000ms: `cy.select()` failed because it could not find a single `<option>` with value or text matching: `avg_products_base_price`
  1. query_level_monitor_spec.js > schedule component displays as intended > for a cron expression schedule
    1. This test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails.
    2. This flaky behavior is also rarely seen impacting the query_level_monitor_spec.js > schedule component displays as intended > for an interval schedule test.
AssertionError: Timed out retrying after 20000ms: Expected to find element: `[data-test-subj="frequency_field"]`, but never found it.
      at Context.eval (https://search-t-nobody-avjv-e2ealerting-3rb2m37iqureafmtutcxcgypem.eu-west-1.es-staging.amazonaws.com/__cypress/tests?p=cypress/integration/plugins/alerting-dashboards-plugin/query_level_monitor_spec.js:580:10)
  1. bucket_level_monitor_spec.js > Bucket-Level Monitors > can be created > by visual editor seems to be flaky when executed by the 1.3 functional test repo branch. It has failed with the error below.
    1. This seems to relate to the use of {enter} when using cy.type() here. Cypress fails to execute the {enter} press.
      1. The screenshot below helps illustrate this; the enter key press doesn't execute, so the UI doesn't detect that a Time Field has been defined which prevents the subsequent UI from rendering.
      2. A potential solution may involve changing a .blur() call at the end of .type() calls that use {enter}.
AssertionError: Timed out retrying: Expected to find element: `[data-test-subj="addMetricButton"]`, but never found it.
  1. cluster_metrics_monitor_spec.js > can be created > for the Cluster Health API seems to occasionally fail when the API is selected from the dropdown before page finishes loading. This causes the selection to reset which causes the test to fail as it expects to see UI elements for the selection. See image screenshot below

  2. monitors_dashboard_spec.js > Displays expected number of alerts occasionally fails with the following error (example). May be related to a delay in alert ingestion.

AssertionError: Timed out retrying after 20000ms: Expected to find element: `:contains(monitors_dashboard_cypress_query_level)`, but never found it. Queried from element: <tr.euiTableRow>
  1. composite_level_monitor_spec.js > can be edited > by visual editor occasionally fails with the following error (example). Adding a short wait after this cy.visit call (link) helps to alleviate the flakiness; but a more permanent solution should be implemented.
  1) CompositeLevelMonitor
       can be edited
         by visual editor:
     AssertionError: Timed out retrying after 10000ms: Expected to find element: `[data-test-subj="monitors_list_2"]`, but never found it.
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/composite_level_monitor_spec.js:339:10)

Contributor guide