Hi Telerik Support,
Time is approx 4th April 16:35 (BST) is the nuget server down again? Getting the following errors via Azure Devops Pipeline Build service:
2024-04-04T15:35:31.4304504Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (nuget.telerik.com:443) 2024-04-04T15:35:31.4305205Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2024-04-04T15:35:31.4305608Z Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled. 2024-04-04T15:35:31.4306411Z Retrying 'FindPackagesByIdAsyncCore' for source 'https://nuget.telerik.com/nuget/FindPackagesById()?id='System.Text.Json'&semVerLevel=2.0.0'. 2024-04-04T15:35:31.4307061Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (nuget.telerik.com:443) 2024-04-04T15:35:31.4310531Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2024-04-04T15:35:31.4319751Z Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled. 2024-04-04T15:35:31.4320549Z Retrying 'FindPackagesByIdAsyncCore' for source 'https://nuget.telerik.com/nuget/FindPackagesById()?id='System.Threading.Tasks.Extensions'&semVerLevel=2.0.0'. 2024-04-04T15:35:31.4321385Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (nuget.telerik.com:443) 2024-04-04T15:35:31.4321947Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2024-04-04T15:35:31.4322678Z Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled. 2024-04-04T15:35:31.4323610Z Retrying 'FindPackagesByIdAsyncCore' for source 'https://nuget.telerik.com/nuget/FindPackagesById()?id='Microsoft.Extensions.Primitives'&semVerLevel=2.0.0'. 2024-04-04T15:35:31.4327699Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (nuget.telerik.com:443) 2024-04-04T15:35:31.4328578Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2024-04-04T15:35:31.4329145Z Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled. 2024-04-04T15:35:31.4373744Z Retrying 'FindPackagesByIdAsyncCore' for source 'https://nuget.telerik.com/nuget/FindPackagesById()?id='System.Security.Permissions'&semVerLevel=2.0.0'. 2024-04-04T15:35:31.4376200Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (nuget.telerik.com:443) 2024-04-04T15:35:31.4376894Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2024-04-04T15:35:31.4377593Z Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled. 2024-04-04T15:35:31.4378276Z Retrying 'FindPackagesByIdAsyncCore' for source 'https://nuget.telerik.com/nuget/FindPackagesById()?id='Microsoft.NETCore.Platforms'&semVerLevel=2.0.0'. 2024-04-04T15:35:31.4379148Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (nuget.telerik.com:443) 2024-04-04T15:35:31.4379624Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2024-04-04T15:35:31.4379967Z Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled. 2024-04-04T15:35:31.4380593Z Retrying 'FindPackagesByIdAsyncCore' for source 'https://nuget.telerik.com/nuget/FindPackagesById()?id='System.Formats.Asn1'&semVerLevel=2.0.0'. 2024-04-04T15:35:31.4382900Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (nuget.telerik.com:443) 2024-04-04T15:35:31.4383469Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2024-04-04T15:35:31.4383874Z Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled. 2024-04-04T15:35:31.4384448Z Retrying 'FindPackagesByIdAsyncCore' for source 'https://nuget.telerik.com/nuget/FindPackagesById()?id='Microsoft.Extensions.Configuration.Abstractions'&semVerLevel=2.0.0'. 2024-04-04T15:35:31.4385205Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (nuget.telerik.com:443) 2024-04-04T15:35:31.4385715Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2024-04-04T15:35:31.4386223Z Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled. 2024-04-04T15:35:31.4386789Z Retrying 'FindPackagesByIdAsyncCore' for source 'https://nuget.telerik.com/nuget/FindPackagesById()?id='Microsoft.Extensions.FileProviders.Abstractions'&semVerLevel=2.0.0'. 2024-04-04T15:35:31.4387318Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (nuget.telerik.com:443) 2024-04-04T15:35:31.4387821Z A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2024-04-04T15:35:31.4388209Z Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled.
This has happened more than a few times in the last 8 weeks. I hope we can resolve once and for all.
Kind regards,
Guy
1 Answer, 1 is accepted
Hi Guy,
Yes, I can confirm there is an ongoing incident where Azure DevOps agents are affected. You can follow the situation here https://status.telerik.com/ (use the subscribe or subscribe to incident options).
The good news is we finally have enough data to understand the root cause and are in the process of working with the datacenter team to avoid this happening again for the same reasons that affected us on Mar 7, Mar 14, Apr 3, and Apr 4 (today).
In the meantime, I would recommend adding a packageSourceMapping to your nuget.config to avoid excessive requests from your builds to the Telerik server.
<?xml version="1.0" encoding="utf-8"?>
<!-- Helpful docs https://docs.microsoft.com/en-us/nuget/reference/nuget-config-file -->
<configuration>
<packageRestore>
<add key="enabled" value="True" />
<add key="automatic" value="True" />
</packageRestore>
<packageManagement>
<add key="format" value="0" />
<add key="disabled" value="False" />
</packageManagement>
<!-- Learn how to set a package source https://docs.microsoft.com/en-us/nuget/reference/nuget-config-file#packagesources -->
<packageSources>
<clear/>
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
<add key="Telerik_Feed" value="https://nuget.telerik.com/v3/index.json" protocolVersion="3"/>
</packageSources>
<!-- This is used by the Windows build. For the Docker build, a Docker Secret is mounted.-->
<packageSourceCredentials>
<Telerik_Feed>
<add key="Username" value="%TELERIK_USERNAME%" />
<add key="ClearTextPassword" value="%TELERIK_PASSWORD%" />
</Telerik_Feed>
</packageSourceCredentials>
<!-- Learn how package source mapping works https://docs.microsoft.com/en-us/nuget/consume-packages/package-source-mapping?Wt.mc_id=DX_MVP5000553 -->
<packageSourceMapping>
<!-- Use the nuget.org for everything else -->
<packageSource key="nuget.org">
<package pattern="*" />
<!-- These are the only 2 packages on nuget.org that needs to be matched before falling to the next mapping-->
<package pattern="Telerik.FontIcons" />
<package pattern="Telerik.SvgIcons" />
</packageSource>
<!-- This mapping group will use the 'Telerik_Feed' package source for ONLY relevant packages.
It avoids erroneously searching the Telerik server for non-Telerik packages-->
<packageSource key="Telerik_Feed">
<package pattern="JustMock.*" />
<package pattern="Telerik.*" />
<package pattern="Kendo.*" />
</packageSource>
</packageSourceMapping>
</configuration>
Regards,
Lance | Manager Technical Support
Progress Telerik
Love the Telerik and Kendo UI products and believe more people should try them? Invite a fellow developer to become a Progress customer and each of you can get a $50 Amazon gift voucher.
Rank 1
Rank 1
Rank 1
Rank 1
Hi folks, some people will still be affected based on what their build machines are doing. I am already working with Daryn, can the rest of you send me your build host's public IP address in a support ticket?
1. Obtain the public IP of your build server
If you don't already know it, you can execute the following command on your build server: (Invoke-WebRequest -uri "http://ifconfig.me/ip").Content
2. Open a new ticket here https://prgress.co/DevToolsSupport
3. Share the IP address with me and we will temporarily whitelist it to let trusted customers through the protection.
Rank 1
Veteran
Iron
Given the circumstances I dont think an active license should be required.
Hi Nils, having a current support license is definitely not a perquisite to accessing the nuget server. We are vetting that folks have had a license and have a need to use the server. Your account has passed that test with flying colors. I will be in contact, check your email in the next couple minutes.
Rank 1
You should probably know, we've been looking into replacing Telerik - this does NOT help your case LOL.
Rank 1
Rank 2
Another Azure DevOps user. Sometimes it will work, but mostly it does not. Ticket submitted with IP info. One issue here is that IP's are dynamically assigned each time the build pipeline is run, so whitelisting individual IP's won't work. You can download Azure IP Ranges and Service Tags for the Public Cloud from Microsoft: https://www.microsoft.com/en-us/download/details.aspx?id=56519
Hi Everyone,
As Brad mentioned, we did consider using that list, but it includes too many potential other sources of non-dev activity (VMs, etc). Instead, we used these known Azure DevOps IP addresses from the Microsoft documentation. However, it does seem to missing a handful of newer ones, so we are cross checking what is available in the latest ServiceTags data against what we're already using to make sure none of them are problematic.
This should be resolved for most AzDO hosted-agent users soon, as the remainder of the IP subnets will be added, and you should no longer have hosted-agent failures during this event. It might still happen if you have a self-hosted agent, in this case, please let me know.
I know this can be frustrating, rest assured that my team and I are tirelessly working on it (I've personally been focused on it since yesterday 8 AM when the event started). It just unfortunately means that while the networking protection is active, which is an absolute requirement, you'd probably had a failed build and gone looking for this forum post... I understand and empathize (I am a dev myself).
Rank 1
Any updates? You've broken a ton of build processes and the status page hasn't been updated in awhile.
Rank 1
We are experiencing this problem for an on-premise TeamCity build server only, so not Azure DevOps agents. Is there something special we need to be doing to get that server's external IP added to an ACL?
Rank 1
Hi folks, the incident is still ongoing. If you are still affected, the only way to address it is to explicitly let you bypass the protection by whitelisting your build machine's public IP address.
If you already have a support ticket open, please get a list of the IP addresses that you need unblocked. Send those IP addresses to me and my team and we'll get those explicitly whitelisted (open a support ticket, please do not share them publicly here).
Once we have those IPs we will send them over to the datacenter's networking team to be allowed to bypass the system. I completely understand this is disruptive, and it can by annoying to have to figure out the IP addresses that you're using. Overall this is what is needed to keep the feed available for everyone, at least until the incident is over.
Tip: If you don't know the IP address of the build machine, you can add a PowerShell task to your CI script, just before the dotnet restore command. This will show you the IP address of that machine.
(Invoke-WebRequest -uri "http://ifconfig.me/ip").ContentHowever, this isn't foolproof, if you're using shared build systems that can be assigned a random IP from a list, you may need to look at the documentation for that system and get the subnet range they use. For example "X.X.X.0/24"
Fixing the Build Right Now
As for production system that need to get unblocked immediately, such as ops's scenario, you do not need the Telerik server, you can instead use a local package source, here's how
1. Download the NuGet packages for the product you're using. These can be found in the Downloads page of your product (they're the .nupkg files).
Go to Downloads | Your Account (telerik.com)
Select your product
Download the package files (P.S. don't forget any Telerik Document Processing packages that you might be using, they're usually listed towards the bottom of the page)
2. Create a new folder in your source code, at the root level, and name it "MyOfflinePackages"
3. Copy/paste all the .nupkg files you just downloaded in Step 1 into that new folder
4. Edit your nuget.config file to use the "MyOfflinePackages" folder as a package source
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<clear/>
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
<!-- Temporarily comment out the server package source, this will prevent errors -->
<!-- <add key="TelerikServerFeed" value="https://nuget.telerik.com/v3/index.json" protocolVersion="3" /> -->
<!-- TEMPORARY LOCAL PACKAGE SOURCE -->
<add key="local-folder-feed" value="..\MyOfflinePackages" />
</packageSources>
...
</configuration>Note: I used the relative path "..\MyOfflinePackages" in the example. Make sure yours matches where you put the folder in relation the nuget restore command is expecting it. For example, if the project is two levels deep, and you put the folder at the top level, then you need to up two folders, like this "..\..\MyOfflinePackages"
This folder-based package sources works exactly the same way that a HTTP-based feed does and will let you get up and running immediately.
Rank 1
Still experiencing issues,,
Rank 1
Hey, whilst i appreciate all the work you are doing to fix it, it took me days to find this thread. Your status page could use an update
Hi Peter and Infra,
Unfortunately the incident is still ongoing and there is no new information to share. However, Infra's feedback is understood and I just finished putting all our guidance into a single document and updated the status page with a link to the gist => Offline Options for NuGet Server.
As for updating the ACL (aka "whitelist"), you can certainly keep sending us your IP addresses and we are sending them over to the datacenter's networking team. Yes, there are dynamic IP systems (like Azure DevOps), but we are adding those to ACL as we learn which ones they are.
Thank you all for your patience and understanding, rest assured that we are working as fast as we can to keep up with your individual requests.
Rank 1
NOTE TO EVERYONE: We have made changes to the networking protection system and are monitoring to see if it helps. Please feel free to chime in if you are now unblocked.
Simon,
Don't worry about getting the IP address(s), we're seeing limited effect for some teams, depending on what system they're using (e.g., AppVeyor systems just keep getting blocked).
To answer your question, you need to next select the product you want the packages for. It's on the product's Downlaod page where they'll be listed (individually or in bundled in a zip)
Lets use Blazor, for example
1. Go to https://www.telerik.com/account/downloads
2. Select Telerik UI for Blazor (which will bring you here https://www.telerik.com/account/downloads/product-download?product=BLAZOR )
3. Pick the version you want packages for in the drop down
4. Scroll down to see all the packages (click each to download it.):
Note: If it's .nupkg file, you can put them directly into your "offline packages" folder. If it is a ZIP, extract the .nupkg files first.
5. Don't for get the others you may need, scroll down a little further to see the Document Processing packages:
Functionally speaking, these .nupkg files are seen by dotnet restore/build/publish just as if they were coming from a REST API-based feed.
Rank 1
Hi Simon, Ah, I see, thanks for clarifying. Yeah try the UI for Blazor one, they'll be listed at the bottom of the page.
This is because the Telerik Document Processing Libraries are not a separate product, instead we bundle it with all relevant products that can use it.
This has been addressed, please let us know if you experience any additional issues. Thank you for your patience and understanding!
Note: There's still a possibility that your build infrastructure's IP address is not covered, in this case, please let me know.
Good work.
Thanks Lance.
I am having the same issue with TeamCity builds.