diff --git a/docs/reference/api.md b/docs/reference/api.md index d55777d8d9e..c9a038ff573 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -70,6 +70,7 @@ _Appears in:_ | `imagePullPolicy` _[PullPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#pullpolicy-v1-core)_ | ImagePullPolicy optionally overrides the autoscaler container's image pull policy. This override is provided for autoscaler testing and development. | | | | `securityContext` _[SecurityContext](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#securitycontext-v1-core)_ | SecurityContext defines the security options the container should be run with.
If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.
More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ | | | | `idleTimeoutSeconds` _integer_ | IdleTimeoutSeconds is the number of seconds to wait before scaling down a worker pod which is not using Ray resources.
Defaults to 60 (one minute). It is not read by the KubeRay operator but by the Ray autoscaler. | | | +| `ttlSecondsAfterIdle` _integer_ | TTLSecondsAfterIdle is the number of seconds to wait before deleting an idle RayCluster.
The Ray autoscaler observes cluster idleness and reports the IdleTTLExpired status condition. The KubeRay operator deletes the RayCluster when the condition is true. | | | | `upscalingMode` _[UpscalingMode](#upscalingmode)_ | UpscalingMode is "Conservative", "Default", or "Aggressive."
Conservative: Upscaling is rate-limited; the number of pending worker pods is at most the size of the Ray cluster.
Default: Upscaling is not rate-limited.
Aggressive: An alias for Default; upscaling is not rate-limited.
It is not read by the KubeRay operator but by the Ray autoscaler. | | Enum: [Default Aggressive Conservative]
| | `version` _[AutoscalerVersion](#autoscalerversion)_ | Version is the version of the Ray autoscaler.
Setting this to v1 will explicitly use autoscaler v1.
Setting this to v2 will explicitly use autoscaler v2.
If this isn't set, the Ray version determines the autoscaler version.
In Ray 2.47.0 and later, the default autoscaler version is v2. It's v1 before that. | | Enum: [v1 v2]
| | `env` _[EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#envvar-v1-core) array_ | Optional list of environment variables to set in the autoscaler container. | | | @@ -665,6 +666,7 @@ _Appears in:_ | `imagePullPolicy` _[PullPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#pullpolicy-v1-core)_ | ImagePullPolicy optionally overrides the autoscaler container's image pull policy. This override is provided for autoscaler testing and development. | | | | `securityContext` _[SecurityContext](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#securitycontext-v1-core)_ | SecurityContext defines the security options the container should be run with.
If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.
More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ | | | | `idleTimeoutSeconds` _integer_ | IdleTimeoutSeconds is the number of seconds to wait before scaling down a worker pod which is not using Ray resources.
Defaults to 60 (one minute). It is not read by the KubeRay operator but by the Ray autoscaler. | | | +| `ttlSecondsAfterIdle` _integer_ | TTLSecondsAfterIdle is the number of seconds to wait before deleting an idle RayCluster.
The Ray autoscaler observes cluster idleness and reports the IdleTTLExpired status condition. The KubeRay operator deletes the RayCluster when the condition is true. | | | | `upscalingMode` _[UpscalingMode](#upscalingmode)_ | UpscalingMode is "Conservative", "Default", or "Aggressive."
Conservative: Upscaling is rate-limited; the number of pending worker pods is at most the size of the Ray cluster.
Default: Upscaling is not rate-limited.
Aggressive: An alias for Default; upscaling is not rate-limited.
It is not read by the KubeRay operator but by the Ray autoscaler. | | Enum: [Default Aggressive Conservative]
| | `env` _[EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#envvar-v1-core) array_ | Optional list of environment variables to set in the autoscaler container. | | | | `envFrom` _[EnvFromSource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#envfromsource-v1-core) array_ | Optional list of sources to populate environment variables in the autoscaler container. | | | diff --git a/helm-chart/kuberay-operator/crds/ray.io_rayclusters.yaml b/helm-chart/kuberay-operator/crds/ray.io_rayclusters.yaml index 1962151dceb..31c409a84ca 100644 --- a/helm-chart/kuberay-operator/crds/ray.io_rayclusters.yaml +++ b/helm-chart/kuberay-operator/crds/ray.io_rayclusters.yaml @@ -296,6 +296,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default @@ -8811,6 +8814,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default diff --git a/helm-chart/kuberay-operator/crds/ray.io_raycronjobs.yaml b/helm-chart/kuberay-operator/crds/ray.io_raycronjobs.yaml index 275aaede81c..ba3a6eefdda 100644 --- a/helm-chart/kuberay-operator/crds/ray.io_raycronjobs.yaml +++ b/helm-chart/kuberay-operator/crds/ray.io_raycronjobs.yaml @@ -387,6 +387,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default diff --git a/helm-chart/kuberay-operator/crds/ray.io_rayjobs.yaml b/helm-chart/kuberay-operator/crds/ray.io_rayjobs.yaml index 0974f37c9e3..8fc08eb1654 100644 --- a/helm-chart/kuberay-operator/crds/ray.io_rayjobs.yaml +++ b/helm-chart/kuberay-operator/crds/ray.io_rayjobs.yaml @@ -391,6 +391,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default @@ -12773,6 +12776,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default diff --git a/helm-chart/kuberay-operator/crds/ray.io_rayservices.yaml b/helm-chart/kuberay-operator/crds/ray.io_rayservices.yaml index 602bfdfb61a..1edd673a5a4 100644 --- a/helm-chart/kuberay-operator/crds/ray.io_rayservices.yaml +++ b/helm-chart/kuberay-operator/crds/ray.io_rayservices.yaml @@ -284,6 +284,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default @@ -9230,6 +9233,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default diff --git a/helm-chart/ray-cluster/tests/raycluster_test.yaml b/helm-chart/ray-cluster/tests/raycluster_test.yaml index 46958e22ad2..14729da9ad9 100644 --- a/helm-chart/ray-cluster/tests/raycluster_test.yaml +++ b/helm-chart/ray-cluster/tests/raycluster_test.yaml @@ -40,6 +40,7 @@ tests: autoscalerOptions: upscalingMode: Default idleTimeoutSeconds: 60 + ttlSecondsAfterIdle: 1800 imagePullPolicy: IfNotPresent env: - name: ENV_KEY @@ -71,6 +72,7 @@ tests: value: upscalingMode: Default idleTimeoutSeconds: 60 + ttlSecondsAfterIdle: 1800 imagePullPolicy: IfNotPresent env: - name: ENV_KEY diff --git a/helm-chart/ray-cluster/values.yaml b/helm-chart/ray-cluster/values.yaml index 9a4b016938e..7bc246b3b1a 100644 --- a/helm-chart/ray-cluster/values.yaml +++ b/helm-chart/ray-cluster/values.yaml @@ -72,6 +72,8 @@ head: # upscalingMode: Default # idleTimeoutSeconds is the number of seconds to wait before scaling down a worker pod which is not using Ray resources. # idleTimeoutSeconds: 60 + # ttlSecondsAfterIdle is the number of seconds to wait before deleting an idle RayCluster. + # ttlSecondsAfterIdle: 1800 # imagePullPolicy optionally overrides the autoscaler container's default image pull policy (IfNotPresent). # imagePullPolicy: IfNotPresent # Optionally specify the autoscaler container's securityContext. diff --git a/ray-operator/apis/ray/v1/raycluster_types.go b/ray-operator/apis/ray/v1/raycluster_types.go index c2d759425fa..568309be2b2 100644 --- a/ray-operator/apis/ray/v1/raycluster_types.go +++ b/ray-operator/apis/ray/v1/raycluster_types.go @@ -227,6 +227,10 @@ type AutoscalerOptions struct { // Defaults to 60 (one minute). It is not read by the KubeRay operator but by the Ray autoscaler. // +optional IdleTimeoutSeconds *int32 `json:"idleTimeoutSeconds,omitempty"` + // TTLSecondsAfterIdle is the number of seconds to wait before deleting an idle RayCluster. + // The Ray autoscaler observes cluster idleness and reports the IdleTTLExpired status condition. The KubeRay operator deletes the RayCluster when the condition is true. + // +optional + TTLSecondsAfterIdle *int32 `json:"ttlSecondsAfterIdle,omitempty"` // UpscalingMode is "Conservative", "Default", or "Aggressive." // Conservative: Upscaling is rate-limited; the number of pending worker pods is at most the size of the Ray cluster. // Default: Upscaling is not rate-limited. @@ -371,6 +375,8 @@ const ( RayClusterSuspending RayClusterConditionType = "RayClusterSuspending" // RayClusterSuspended is set to true when all Pods belonging to a suspending RayCluster are deleted. Note that RayClusterSuspending and RayClusterSuspended cannot both be true at the same time. RayClusterSuspended RayClusterConditionType = "RayClusterSuspended" + // IdleTTLExpired is set to true by the Ray autoscaler when the cluster has been idle longer than spec.autoscalerOptions.ttlSecondsAfterIdle. + IdleTTLExpired RayClusterConditionType = "IdleTTLExpired" ) // HeadInfo gives info about head diff --git a/ray-operator/apis/ray/v1/zz_generated.deepcopy.go b/ray-operator/apis/ray/v1/zz_generated.deepcopy.go index fcfb31ce811..5c8770efc35 100644 --- a/ray-operator/apis/ray/v1/zz_generated.deepcopy.go +++ b/ray-operator/apis/ray/v1/zz_generated.deepcopy.go @@ -85,6 +85,11 @@ func (in *AutoscalerOptions) DeepCopyInto(out *AutoscalerOptions) { *out = new(int32) **out = **in } + if in.TTLSecondsAfterIdle != nil { + in, out := &in.TTLSecondsAfterIdle, &out.TTLSecondsAfterIdle + *out = new(int32) + **out = **in + } if in.UpscalingMode != nil { in, out := &in.UpscalingMode, &out.UpscalingMode *out = new(UpscalingMode) diff --git a/ray-operator/apis/ray/v1alpha1/raycluster_types.go b/ray-operator/apis/ray/v1alpha1/raycluster_types.go index 48e6c88c7dc..e523c90108d 100644 --- a/ray-operator/apis/ray/v1alpha1/raycluster_types.go +++ b/ray-operator/apis/ray/v1alpha1/raycluster_types.go @@ -84,6 +84,9 @@ type AutoscalerOptions struct { // IdleTimeoutSeconds is the number of seconds to wait before scaling down a worker pod which is not using Ray resources. // Defaults to 60 (one minute). It is not read by the KubeRay operator but by the Ray autoscaler. IdleTimeoutSeconds *int32 `json:"idleTimeoutSeconds,omitempty"` + // TTLSecondsAfterIdle is the number of seconds to wait before deleting an idle RayCluster. + // The Ray autoscaler observes cluster idleness and reports the IdleTTLExpired status condition. The KubeRay operator deletes the RayCluster when the condition is true. + TTLSecondsAfterIdle *int32 `json:"ttlSecondsAfterIdle,omitempty"` // UpscalingMode is "Conservative", "Default", or "Aggressive." // Conservative: Upscaling is rate-limited; the number of pending worker pods is at most the size of the Ray cluster. // Default: Upscaling is not rate-limited. diff --git a/ray-operator/apis/ray/v1alpha1/zz_generated.deepcopy.go b/ray-operator/apis/ray/v1alpha1/zz_generated.deepcopy.go index 8619f9470b5..a520fede546 100644 --- a/ray-operator/apis/ray/v1alpha1/zz_generated.deepcopy.go +++ b/ray-operator/apis/ray/v1alpha1/zz_generated.deepcopy.go @@ -63,6 +63,11 @@ func (in *AutoscalerOptions) DeepCopyInto(out *AutoscalerOptions) { *out = new(int32) **out = **in } + if in.TTLSecondsAfterIdle != nil { + in, out := &in.TTLSecondsAfterIdle, &out.TTLSecondsAfterIdle + *out = new(int32) + **out = **in + } if in.UpscalingMode != nil { in, out := &in.UpscalingMode, &out.UpscalingMode *out = new(UpscalingMode) diff --git a/ray-operator/config/crd/bases/ray.io_rayclusters.yaml b/ray-operator/config/crd/bases/ray.io_rayclusters.yaml index 1962151dceb..31c409a84ca 100644 --- a/ray-operator/config/crd/bases/ray.io_rayclusters.yaml +++ b/ray-operator/config/crd/bases/ray.io_rayclusters.yaml @@ -296,6 +296,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default @@ -8811,6 +8814,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default diff --git a/ray-operator/config/crd/bases/ray.io_raycronjobs.yaml b/ray-operator/config/crd/bases/ray.io_raycronjobs.yaml index 275aaede81c..ba3a6eefdda 100644 --- a/ray-operator/config/crd/bases/ray.io_raycronjobs.yaml +++ b/ray-operator/config/crd/bases/ray.io_raycronjobs.yaml @@ -387,6 +387,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default diff --git a/ray-operator/config/crd/bases/ray.io_rayjobs.yaml b/ray-operator/config/crd/bases/ray.io_rayjobs.yaml index 0974f37c9e3..8fc08eb1654 100644 --- a/ray-operator/config/crd/bases/ray.io_rayjobs.yaml +++ b/ray-operator/config/crd/bases/ray.io_rayjobs.yaml @@ -391,6 +391,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default @@ -12773,6 +12776,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default diff --git a/ray-operator/config/crd/bases/ray.io_rayservices.yaml b/ray-operator/config/crd/bases/ray.io_rayservices.yaml index 602bfdfb61a..1edd673a5a4 100644 --- a/ray-operator/config/crd/bases/ray.io_rayservices.yaml +++ b/ray-operator/config/crd/bases/ray.io_rayservices.yaml @@ -284,6 +284,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default @@ -9230,6 +9233,9 @@ spec: type: string type: object type: object + ttlSecondsAfterIdle: + format: int32 + type: integer upscalingMode: enum: - Default diff --git a/ray-operator/controllers/ray/common/rbac.go b/ray-operator/controllers/ray/common/rbac.go index 3dbecded50d..f5e318310ce 100644 --- a/ray-operator/controllers/ray/common/rbac.go +++ b/ray-operator/controllers/ray/common/rbac.go @@ -54,6 +54,11 @@ func BuildRole(cluster *rayv1.RayCluster) (*rbacv1.Role, error) { Resources: []string{"rayclusters"}, Verbs: []string{"get", "patch"}, }, + { + APIGroups: []string{"ray.io"}, + Resources: []string{"rayclusters/status"}, + Verbs: []string{"patch"}, + }, }, } diff --git a/ray-operator/controllers/ray/common/rbac_test.go b/ray-operator/controllers/ray/common/rbac_test.go index 253ee6b318c..63759187698 100644 --- a/ray-operator/controllers/ray/common/rbac_test.go +++ b/ray-operator/controllers/ray/common/rbac_test.go @@ -6,6 +6,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" rayv1 "github.com/ray-project/kuberay/ray-operator/apis/ray/v1" @@ -85,3 +86,21 @@ func TestBuildRoleBindingSubjectAndRoleRefName(t *testing.T) { }) } } + +func TestBuildRoleAllowsAutoscalerToPatchRayClusterStatus(t *testing.T) { + cluster := &rayv1.RayCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "raycluster-sample", + Namespace: "default", + }, + } + + role, err := BuildRole(cluster) + require.NoError(t, err) + + assert.Contains(t, role.Rules, rbacv1.PolicyRule{ + APIGroups: []string{"ray.io"}, + Resources: []string{"rayclusters/status"}, + Verbs: []string{"patch"}, + }) +} diff --git a/ray-operator/controllers/ray/raycluster_controller.go b/ray-operator/controllers/ray/raycluster_controller.go index 033a2395894..bdfa50dce83 100644 --- a/ray-operator/controllers/ray/raycluster_controller.go +++ b/ray-operator/controllers/ray/raycluster_controller.go @@ -293,6 +293,15 @@ func (r *RayClusterReconciler) rayClusterReconcile(ctx context.Context, instance return ctrl.Result{}, nil } + if cond := meta.FindStatusCondition(instance.Status.Conditions, string(rayv1.IdleTTLExpired)); cond != nil && cond.Status == metav1.ConditionTrue && isIdleTTLTerminationEnabled(instance) { + logger.Info("Deleting RayCluster because the idle TTL has expired", "condition", cond) + r.Recorder.Eventf(instance, corev1.EventTypeNormal, string(rayv1.IdleTTLExpired), "%s", cond.Message) + if err := r.Delete(ctx, instance); err != nil { + return ctrl.Result{RequeueAfter: DefaultRequeueDuration}, client.IgnoreNotFound(err) + } + return ctrl.Result{}, nil + } + reconcileFuncs := []reconcileFunc{ r.reconcileAutoscalerServiceAccount, r.reconcileAutoscalerRole, @@ -355,6 +364,13 @@ func (r *RayClusterReconciler) rayClusterReconcile(ctx context.Context, instance return ctrl.Result{RequeueAfter: time.Duration(requeueAfterSeconds) * time.Second}, nil } +func isIdleTTLTerminationEnabled(instance *rayv1.RayCluster) bool { + return instance != nil && + utils.IsAutoscalingEnabled(&instance.Spec) && + instance.Spec.AutoscalerOptions != nil && + instance.Spec.AutoscalerOptions.TTLSecondsAfterIdle != nil +} + func (r *RayClusterReconciler) reconcileAuthSecret(ctx context.Context, instance *rayv1.RayCluster) error { logger := ctrl.LoggerFrom(ctx) logger.Info("Reconciling Auth") diff --git a/ray-operator/controllers/ray/raycluster_controller_unit_test.go b/ray-operator/controllers/ray/raycluster_controller_unit_test.go index bd1d78582e2..8452688046a 100644 --- a/ray-operator/controllers/ray/raycluster_controller_unit_test.go +++ b/ray-operator/controllers/ray/raycluster_controller_unit_test.go @@ -3438,6 +3438,101 @@ func Test_ReconcileManagedBy(t *testing.T) { } } +func Test_ReconcileDeletesRayClusterWhenIdleTTLExpired(t *testing.T) { + setupTest(t) + ctx := context.Background() + newScheme := runtime.NewScheme() + _ = rayv1.AddToScheme(newScheme) + _ = corev1.AddToScheme(newScheme) + + cluster := testRayCluster.DeepCopy() + cluster.Spec.EnableInTreeAutoscaling = new(true) + cluster.Spec.AutoscalerOptions = &rayv1.AutoscalerOptions{ + TTLSecondsAfterIdle: new(int32(1800)), + } + cluster.Status.Conditions = []metav1.Condition{ + { + Type: string(rayv1.IdleTTLExpired), + Status: metav1.ConditionTrue, + Reason: "ClusterIdlePastTTL", + Message: "Cluster idle for 1823s exceeds ttlSecondsAfterIdle=1800s", + }, + } + + fakeClient := clientFake.NewClientBuilder(). + WithScheme(newScheme). + WithObjects(cluster). + WithStatusSubresource(cluster). + Build() + recorder := record.NewFakeRecorder(1) + reconciler := &RayClusterReconciler{ + Client: fakeClient, + Recorder: recorder, + Scheme: newScheme, + rayClusterScaleExpectation: expectations.NewRayClusterScaleExpectation(fakeClient), + } + + result, err := reconciler.rayClusterReconcile(ctx, cluster) + require.NoError(t, err) + assert.Zero(t, result.RequeueAfter) + + deletedCluster := &rayv1.RayCluster{} + err = fakeClient.Get(ctx, types.NamespacedName{Name: cluster.Name, Namespace: cluster.Namespace}, deletedCluster) + assert.True(t, k8serrors.IsNotFound(err)) + + event := <-recorder.Events + assert.Contains(t, event, string(rayv1.IdleTTLExpired)) + assert.Contains(t, event, "Cluster idle for 1823s exceeds ttlSecondsAfterIdle=1800s") +} + +func TestIsIdleTTLTerminationEnabled(t *testing.T) { + testCases := map[string]struct { + mutate func(*rayv1.RayCluster) + enabled bool + }{ + "enabled": { + mutate: func(cluster *rayv1.RayCluster) { + cluster.Spec.EnableInTreeAutoscaling = new(true) + cluster.Spec.AutoscalerOptions = &rayv1.AutoscalerOptions{ + TTLSecondsAfterIdle: new(int32(1800)), + } + }, + enabled: true, + }, + "autoscaling disabled": { + mutate: func(cluster *rayv1.RayCluster) { + cluster.Spec.EnableInTreeAutoscaling = new(false) + cluster.Spec.AutoscalerOptions = &rayv1.AutoscalerOptions{ + TTLSecondsAfterIdle: new(int32(1800)), + } + }, + enabled: false, + }, + "ttl unset": { + mutate: func(cluster *rayv1.RayCluster) { + cluster.Spec.EnableInTreeAutoscaling = new(true) + cluster.Spec.AutoscalerOptions = &rayv1.AutoscalerOptions{} + }, + enabled: false, + }, + "autoscaler options unset": { + mutate: func(cluster *rayv1.RayCluster) { + cluster.Spec.EnableInTreeAutoscaling = new(true) + cluster.Spec.AutoscalerOptions = nil + }, + enabled: false, + }, + } + + for name, tc := range testCases { + t.Run(name, func(t *testing.T) { + cluster := testRayCluster.DeepCopy() + tc.mutate(cluster) + assert.Equal(t, tc.enabled, isIdleTTLTerminationEnabled(cluster)) + }) + } +} + func TestEmitRayClusterProvisionedDuration(t *testing.T) { clusterName := "test-ray-cluster" clusterNamespace := "default" diff --git a/ray-operator/controllers/ray/utils/validation.go b/ray-operator/controllers/ray/utils/validation.go index 41b5855a4ee..c670c6205e7 100644 --- a/ray-operator/controllers/ray/utils/validation.go +++ b/ray-operator/controllers/ray/utils/validation.go @@ -234,6 +234,14 @@ func ValidateRayClusterSpec(spec *rayv1.RayClusterSpec, annotations map[string]s return fmt.Errorf("autoscalerOptions.idleTimeoutSeconds must be non-negative, got %d", *spec.AutoscalerOptions.IdleTimeoutSeconds) } } + if spec.AutoscalerOptions != nil && spec.AutoscalerOptions.TTLSecondsAfterIdle != nil { + if *spec.AutoscalerOptions.TTLSecondsAfterIdle < 0 { + return fmt.Errorf("autoscalerOptions.ttlSecondsAfterIdle must be non-negative, got %d", *spec.AutoscalerOptions.TTLSecondsAfterIdle) + } + if !isAutoscalingEnabled { + return fmt.Errorf("autoscalerOptions.ttlSecondsAfterIdle requires enableInTreeAutoscaling to be true") + } + } if IsAuthEnabled(spec) { if spec.RayVersion == "" { diff --git a/ray-operator/controllers/ray/utils/validation_test.go b/ray-operator/controllers/ray/utils/validation_test.go index 9a481176996..ba5828eb81a 100644 --- a/ray-operator/controllers/ray/utils/validation_test.go +++ b/ray-operator/controllers/ray/utils/validation_test.go @@ -2268,6 +2268,47 @@ func TestValidateRayClusterSpec_IdleTimeoutSeconds(t *testing.T) { }(), expectedErr: "autoscalerOptions.idleTimeoutSeconds must be non-negative, got -10", }, + "Valid: AutoscalerOptions with ttlSecondsAfterIdle": { + spec: func() rayv1.RayClusterSpec { + s := createSpec() + s.AutoscalerOptions = &rayv1.AutoscalerOptions{ + TTLSecondsAfterIdle: new(int32(1800)), + } + return s + }(), + expectedErr: "", + }, + "Valid: AutoscalerOptions with zero ttlSecondsAfterIdle": { + spec: func() rayv1.RayClusterSpec { + s := createSpec() + s.AutoscalerOptions = &rayv1.AutoscalerOptions{ + TTLSecondsAfterIdle: new(int32(0)), + } + return s + }(), + expectedErr: "", + }, + "Invalid: AutoscalerOptions with negative ttlSecondsAfterIdle": { + spec: func() rayv1.RayClusterSpec { + s := createSpec() + s.AutoscalerOptions = &rayv1.AutoscalerOptions{ + TTLSecondsAfterIdle: new(int32(-10)), + } + return s + }(), + expectedErr: "autoscalerOptions.ttlSecondsAfterIdle must be non-negative, got -10", + }, + "Invalid: AutoscalerOptions with ttlSecondsAfterIdle and autoscaling disabled": { + spec: func() rayv1.RayClusterSpec { + s := createSpec() + s.EnableInTreeAutoscaling = new(false) + s.AutoscalerOptions = &rayv1.AutoscalerOptions{ + TTLSecondsAfterIdle: new(int32(1800)), + } + return s + }(), + expectedErr: "autoscalerOptions.ttlSecondsAfterIdle requires enableInTreeAutoscaling to be true", + }, "Valid: AutoscalerOptions without idleTimeoutSeconds": { spec: func() rayv1.RayClusterSpec { s := createSpec() diff --git a/ray-operator/pkg/client/applyconfiguration/ray/v1/autoscaleroptions.go b/ray-operator/pkg/client/applyconfiguration/ray/v1/autoscaleroptions.go index fa86b33188c..0362a6a5cf2 100644 --- a/ray-operator/pkg/client/applyconfiguration/ray/v1/autoscaleroptions.go +++ b/ray-operator/pkg/client/applyconfiguration/ray/v1/autoscaleroptions.go @@ -26,6 +26,9 @@ type AutoscalerOptionsApplyConfiguration struct { // IdleTimeoutSeconds is the number of seconds to wait before scaling down a worker pod which is not using Ray resources. // Defaults to 60 (one minute). It is not read by the KubeRay operator but by the Ray autoscaler. IdleTimeoutSeconds *int32 `json:"idleTimeoutSeconds,omitempty"` + // TTLSecondsAfterIdle is the number of seconds to wait before deleting an idle RayCluster. + // The Ray autoscaler observes cluster idleness and reports the IdleTTLExpired status condition. The KubeRay operator deletes the RayCluster when the condition is true. + TTLSecondsAfterIdle *int32 `json:"ttlSecondsAfterIdle,omitempty"` // UpscalingMode is "Conservative", "Default", or "Aggressive." // Conservative: Upscaling is rate-limited; the number of pending worker pods is at most the size of the Ray cluster. // Default: Upscaling is not rate-limited. @@ -92,6 +95,14 @@ func (b *AutoscalerOptionsApplyConfiguration) WithIdleTimeoutSeconds(value int32 return b } +// WithTTLSecondsAfterIdle sets the TTLSecondsAfterIdle field in the declarative configuration to the given value +// and returns the receiver, so that objects can be built by chaining "With" function invocations. +// If called multiple times, the TTLSecondsAfterIdle field is set to the value of the last call. +func (b *AutoscalerOptionsApplyConfiguration) WithTTLSecondsAfterIdle(value int32) *AutoscalerOptionsApplyConfiguration { + b.TTLSecondsAfterIdle = &value + return b +} + // WithUpscalingMode sets the UpscalingMode field in the declarative configuration to the given value // and returns the receiver, so that objects can be built by chaining "With" function invocations. // If called multiple times, the UpscalingMode field is set to the value of the last call.