Issue Alerts
⚠️ Resolved: App Builder Domains Not Provisioned in UK (November 13th, 2024)
Issue Resolved
Date: Friday, November 13th, 2024 Time: 7:38 PM EDT
We were performing upgrades across all Rewst instances to how we provision domains for live sites. The upgrades were successful in all of our instances, but failed in prod US at 6:33 PM EST. We were troubleshooting for 50 minutes and managed to roll back at 7:23PM bringing rew.st domains back up. There was additional work needed to get custom domains back, which was completed at 7:38PM.
Issue Identified
Date: Friday, November 13th, 2024 Time: 6:33 PM EDT
App Builder domains were not being provisioned in UK leading to live sites not being available.
Updates
For the latest information, please refer to https://status.rewst.io/
Please contact our support team if you have any questions or need further assistance.
Discord - The ROC is always available here: https://discord.gg/rewst
Create a Ticket - E-mail the team and someone will be in touch ASAP!
⚠️ Resolved: Platform Disruption to Workflow Processes for US Customers (November 6th, 2024)
Date: Wednesday, November 6th, 2024
Time: 10:53 AM - 11:40AM EST
On November 6, 2024, from 10:53 AM EST to 11:40 AM EST, an unintended disruption occurred in our production environment, impacting workflow processing for our US customers. While working towards a resolution, there were failures of workflows running at the time. This temporarily affected real-time processing across parts of our application. The issue was resolved and workflows are running as normal.
Thank you for your understanding as we work to strengthen our safeguards. We are committed to maintaining the reliability and resilience of our platform and will continue to enhance our processes to prevent such incidents in the future.
Updates
For the latest information, please refer to https://status.rewst.io/
Please contact our support team if you have any questions or need further assistance.
Discord - The ROC is always available here: https://discord.gg/rewst
Create a Ticket - E-mail the team and someone will be in touch ASAP!
⚠️ Resolved: Rewst Experiencing Performance Degradation for US Region (November 1st, 2024)
Issue Resolved
Date: Friday, November 1st, 2024 Time: 4:30 PM EDT
After further investigation, it was discovered that a previous settings change intended to improve performance was causing timeouts because of the large number of connections in the environment. Upon reverting the setting, performance returned back to normal around 4:30PM EDT.
Issue Identified
Date: Friday, November 1st, 2024
Time: 11:58 AM EST
ReOn Friday, November 1, 2024 beginning at 10:23AM EDT, the workflow engine began exhibiting slow performance while executing tasks. Our platform team was alerted when the pending task queue grew out of normal boundaries and began investigating the problem.
Updates
Time: 2:00 PM EST Investigation showed that increasing the number of worker nodes had an adverse affect on workflow processing, pointing to a potential issue with the message queue system experiencing timeouts. To stabilize the task queues, the team adjusted the number of worker nodes until processing normalized, while continuing to troubleshoot the root cause. By 2PM EDT, the task processing rate was normalized, although still slower than normal.
For the latest information, please refer to https://status.rewst.io/
Please contact our support team if you have any questions or need further assistance.
Discord - The ROC is always available here: https://discord.gg/rewst
Create a Ticket - E-mail the team and someone will be in touch ASAP!
⚠️ Resolved: Workflows Failing to Execute in app.rewst.eu (October 31st, 2024)
Date: Thursday, October 31st, 2024
Time: 8:20 UTC / 4:20 AM EDT
October 31st, 2024 at 8:20 UTC Rewst discovered an issue with workflow executions initializing in the European instance. UK Customers were not affected. This was escalated internally for review per the standard process. Rewst traced the issue to a partition issue. This issue was fixed and Rewst is currently operating as normal while we work to publish a long-term fix to ensure this doesn't happen again.
Updates
For the latest information, please refer to https://status.rewst.io/
Please contact our support team if you have any questions or need further assistance.
Discord - The ROC is always available here: https://discord.gg/rewst
Create a Ticket - E-mail the team and someone will be in touch ASAP!
⚠️ Routine System Maintenance (October 26, 2024)
Date: Saturday, October 26, 2024
Time: 3PM to 3:30PM ET
During this period, the system will remain online; however, you may experience temporary performance degradation, such as slower response times. We appreciate your understanding as we work to enhance the performance and reliability of our platform. If you have any concerns or experience significant issues, please reach out to our support team or your CSM.
⚠️ Resolved: Slow User Interface (UI) Load Times (October 3rd, 2024)
Date: Thursday, October 3rd, 2024
Time: 22:00 UTC / 6:00 PM EDT
October 3, 2024 at 22:00 UTC Rewst was notified of slow user interface (UI) load times and reduced accessibility of the platform. This was escalated internally for review per standard process. Rewst traced the issue to a slow-running common request. This resulted in Rewst Forms and UI elements not loading for some users and reduced functionality for other users from 22:00 to 23:15 UTC. Rewst is currently operating as normal while we continue to investigate the root cause of this issue. More information will be provided as it becomes available.
⚠️ Update To Disable Forms (September 30th, 2024)
Issue Identified
Date: Monday, September 30th, 2024
We released an update that corrected an issue where some forms marked as "disabled" were still functioning. If you had disabled form triggers within a workflow, they may now be correctly inactive. We’ve noticed a few customers were using these forms in production, and the update has properly disabled them.
If you’re impacted by this change, simply re-enable the form in your trigger settings.
⚠️ Resolved: Lost Form Access (September 12th, 2024)
Issue Identified
Date: Thursday, September 12, 2024 Time: 11:17 PM EDT
An update aimed at improving load times for custom forms was implemented at 3 PM EDT. This update unintentionally caused some users to lose access to forms.
Issue Resolved
Date: Thursday, September 12, 2024 Time: 11:55 PM EDT
Following standard procedures, the update was rolled back, and the issue was resolved.
If you have any questions or concerns, please contact the ROC support team via Discord or contact your Customer Success Manager.
Thank you for your understanding and patience as we work to improve our platform!
⚠️ Planned Service Maintenance Notification (August 14th, 2024)
Date: Friday, August 16th, 2024 Time: 6:00 PM - 7:00 PM EST
We will be performing a scheduled update to our platform to prepare for some awesome new features that should be released soon! While most users won't notice any disruption, the following features will be temporarily unavailable for approximately 10 to 20 minutes during this period:
Loading Action Options
Cloning, Syncing, Exporting, and Importing
RoboRewsty functionality
Jinja Rendering and Live Editor
App Platform page rendering
Unpacking Crates
Processing of workflow completion handlers (pending tasks will resume after the update)
⚠️ Please note that all other workflows and webhooks will operate normally during this time. ⚠️
If you have any questions or concerns, please contact the ROC support team via Discord or contact your Customer Success Manager.
Thank you for your understanding and patience as we work to improve our platform!
⚠️ Resolved: Update Service Ticket Action Failure (August 7th, 2024)
⚠️ We are pleased to inform you that the issue affecting workflows containing the ConnectWise Update Service Ticket (v1) action has been successfully resolved. Normal functionality has resumed. All services should now be operating as expected. ⚠️
Technical Update for Rewst Users
Issue: We detected an issue affecting workflows that update PSA tickets.
The action affected is: ConnectWise Update Service Ticket (v1). Workflows that contain this action are failing.
This also affects crates that use this action, such as User Onboarding and User Offboarding V1 and V2. Once the recovery process is complete, we will re-run all failed New User Workflows.
Current Status
Our team is actively working on restoring the system from backup, and we expect to resume normal functionality within the next hour.
We appreciate your patience and will provide updates once we have more information.
Updates
Thank you for your understanding and cooperation.
Please contact our support team if you have any questions or need further assistance.
Discord - The ROC is always available here: https://discord.gg/rewst
Create a Ticket - E-mail the team and someone will be in touch ASAP!
⚠️ Resolved: Forms Users Experiencing Login Delays and Timeouts (July 16th, 2024)
⚠️ This issue has been successfully resolved and we are closing this alert. An RCA report will be available upon request. ⚠️
Technical Update for Rewst Users
Issue: Intermittent Delays or Timeouts During Login. We have received reports from users assigned the "Forms" role experiencing intermittent delays or timeouts when logging into Rewst.
Current Status:
Affected Users: Users with the "Forms" role
Symptom: Intermittent delays or timeouts during login
Investigation: Ongoing to determine the root cause and implement a solution
Our team is actively working to identify and resolve the underlying problem. We appreciate your patience and will provide updates once we have more information.
Next Steps:
We are closely monitoring the system.
We are working to identify the root cause.
Optional Workaround: MSPs may temporarily add the Read Only Role to affected users.
Updates:
Thank you for your understanding and cooperation.
For the latest information, please refer to https://status.rewst.io/
Please contact our support team if you have any questions or need further assistance.
Discord - The ROC is always available here: https://discord.gg/rewst
Create a Ticket - E-mail the team and someone will be in touch ASAP!
⚠️ The new UI enhancement highlights any Rewst Organization linked to multiple CSP Customers (July 3rd, 2024)
Error Indication
Problem Scenario
If a Rewst Organization is mapped to more than one CSP Customer, it causes ambiguity in workflow execution. Since workflows can only follow the most recent mapping, this can result in:
Workflows failing to execute correctly.
Unexpected actions being performed on the wrong CSP Customer.
Solution Steps
To resolve this issue, ensure that each Rewst Organization is mapped to a single CSP Customer. Follow these guidelines:
Single Mapping Per Rewst Organization:
Each Rewst Organization should be linked to only one CSP Customer.
Example: Rewst Organization A → CSP Customer 1.
Multiple Rewst Organizations to One CSP Customer:
You can map the same CSP Customer to multiple Rewst Organizations.
Example: Rewst Organization A, Rewst Organization B, and Rewst Organization C can all be mapped to CSP Customer 1 if they represent the same underlying CSP customer.
Avoid Multiple CSP Customers for One Rewst Organization:
The new error message ensures a Rewst Organization is not linked to multiple CSP Customers to avoid conflicts.
Example: Rewst Organization A should not be mapped to CSP Customer 1, CSP Customer 2, and CSP Customer 3.
Example Explanation
In this example, the Rewst Organization "The Kewp" is mapped to two different CSP Customers: "thekewp" and "Rewst Development." This setup triggers an error message indicating a conflict: "One or more organizations are linked to multiple customers. This will cause errors when attempting to run Microsoft actions on these organizations."
Resolution Steps
To resolve these errors, you need to ensure that "The Kewp" Rewst Organization is mapped to only one CSP Customer. Follow these steps:
Map Rewst Organization "The Kewp" → CSP Customer "thekewp".
Remove the relationship mapping between Rewst Organization "The Kewp" and CSP Customer "Rewst Development".
After making these changes, save your results to ensure the workflow executes correctly without conflicts.
Practical Mapping Example
Here’s how to correctly set up your mappings:
Correct Setup Example 1:
Rewst Organization A → CSP Customer 1.
Rewst Organization B → CSP Customer 2.
Rewst Organization C → CSP Customer 3.
Correct Setup Example 2:
Rewst Organization A → CSP Customer 1.
Rewst Organization B → CSP Customer 1.
Rewst Organization C → CSP Customer 1.
Incorrect Setup (Conflict):
Rewst Organization A → CSP Customer 1, CSP Customer 2, CSP Customer 3.
By following these mapping rules, you can ensure that your workflows are executed correctly and directed to the appropriate CSP Customer, preventing any unexpected issues.
Last updated
