Update `New-ReferenceAssembly` and `New-UnifiedNugetPackage` to generate reference assembly for `Microsoft.PowerShell.Commands.Utility.dll` and properly deploy it for `Microsoft.PowerShell.Commands.Utility` NuGet package and `Microsoft.PowerShell.SDK` NuGet package.
An incremental step to fix, eventually, #8121
Update `New-ReferenceAssembly` and `New-UnifiedNugetPackage` to generate reference assembly for `Microsoft.PowerShell.Commands.Utility.dll` and properly deploy it for `Microsoft.PowerShell.Commands.Utility` NuGet package and `Microsoft.PowerShell.SDK` NuGet package.
An incremental step to fix, eventually, #8121
We have the public API `JsonObject.ConvertFromJson` to convert from JSON string in the PowerShell context. It would be good to have a public API for conversion to JSON. This PR refactors the `ConvertTo-Json` cmdlet to move the core implementation to `JsonObject.ConvertToJson`, and make `ConvertTo-Json` call that public method.
This would help the Azure Function PowerShell worker. Currently, we depends on [calling the cmdlet](https://github.com/Azure/azure-functions-powershell-worker/blob/729710d2595b4b96abd4fa5a5547fe880e98bfa6/src/PowerShell/PowerShellManager.cs#L198-L205) to convert object to JSON which is expensive. Once we have the public method `JsonObject.ConvertToJson` exposed, we can call the API directly to avoid a command invocation.
# Conflicts:
# test/Test.Common.props
We have the public API `JsonObject.ConvertFromJson` to convert from JSON string in the PowerShell context. It would be good to have a public API for conversion to JSON. This PR refactors the `ConvertTo-Json` cmdlet to move the core implementation to `JsonObject.ConvertToJson`, and make `ConvertTo-Json` call that public method.
This would help the Azure Function PowerShell worker. Currently, we depends on [calling the cmdlet](https://github.com/Azure/azure-functions-powershell-worker/blob/729710d2595b4b96abd4fa5a5547fe880e98bfa6/src/PowerShell/PowerShellManager.cs#L198-L205) to convert object to JSON which is expensive. Once we have the public method `JsonObject.ConvertToJson` exposed, we can call the API directly to avoid a command invocation.
The last two commits to this file "broke" both "Developing and contributing"(link is missing) and "Support" (don't know what it didn't format correctly.
This commit fixes both cases.
Major changes are:
- Rename `s_wasSystemPolicyDebugPolicy` to `s_allowDebugOverridePolicy` to make it less confusing. Also slightly refactor `HelperSecurity.psm1` and `ConstrainedLanguageDebugger.Tests.ps1` to remove unneeded code. There is no functional change in this commit.
- Remove the unneeded static property `IsInbox`, as PowerShell Core won't be shipped in-box with Windows in the foreseeable feature. Even if we do in future, we won't be needing it because Windows PowerShell will probably be gone by that time.
- Update 'BindRunspace' to avoid getting all commands and unneeded method calls.
- Avoid creating a `IsSafeValueVisitor` every time when `IsScriptBlockInFactASafeHashtable` runs.
Major changes are as follows:
- Avoid `SecuritySupport.IsProductBinary` and unnecessary AMSI/suspicious code scan at startup time
- Update `CompiledScriptBlockData.IsProductCode` to avoid unnecessary calls to `IsProductBinary`, which attempts to retrieve catalog signature of the target file.
- Update `PerformSecurityChecks` to skip AMSI and suspicious code scan for the `.psd1` file that contains a safe `HashtableAst` only.
- Use customized `ReadOnlyBag` instead of `ImmutableHashSet` so that we can avoid loading the `System.Collections.Immutable.dll` completely.
- Replace `SHA1` with `CRC32` when generating module analysis cache file name
- This remove the loading of `System.Security.Cryptography.Algorithms.dll` at startup
- Move `ConvertFrom-SddlString` to C# to remove the `Utility.psm1` file.
- Crossgen `Microsoft.ApplicationInsights.dll` and enable tiered compilation
- Even pwsh with crossgen assemblies spends a lot time in jitting at the startup, about `191.6ms` comparing with `24.7ms` for Windows PowerShell.
- Jitting `Microsoft.ApplicationInsights.dll` takes about `51.6ms`.
- By crossgen `Microsoft.ApplicationInsights.dll` and enable tiered compilation, the jitting time drops to about `98.9ms`.
The root cause is that `OutputBuffer` is not cleaned up (set to `null`) in `Stop` and `BeginStop/EndStop` when the powershell instance owns the `OutputBuffer` object. Since the pipeline has been intentionally stopped by the caller, the `OutputBuffer` should be set to `null` as well when it's owned by the PowerShell instance, just like how it's cleaned up in `EndInvoke`.
The type System.IntPtr has an explicit cast operator defined that takes a pointer type System.void* parameter. Now we handle that type properly in GetPSMethodProjectedType.
Support calling methods with ByRef-like type parameters in PowerShell, as long as the argument specified for the parameter can be implicitly/explicitly converted to the ByRef-like type.
We cannot create an instance of a ByRef-like type in PowerShell, but there are types that can be implicitly or explicitly converted to ByRef-like types, such as `T[] -> Span<T>`, `T[] -> ReadOnlySpan<T>`, `String -> ReadOnlySpan<T>`. `ArraySegment<T> -> Span<T>`, `ArraySegment<T> -> ReadOnlySpan<T>`. With changes in this PR, we can call methods with ByRef-like parameter types by providing the arguments of the types that can be cast to the target ByRef-like type.
**What is enabled?**
1. Invoking methods that have ByRef-like parameters, but the return type is not ByRef-like.
2. Invoking constructors with ByRef-like parameters via `[target-type]::new` syntax, but the `target-type` is not ByRef-like.
3. Accessing indexers that have ByRef-like parameters, but the return type is not ByRef-like.
ByRef-like types are supposed to be used on stack only, so we need to fail gracefully when accessing properties, calling methods, or creating objects related to ByRef-like types.
A comment from the existing `Utils.IsComObject` mentions that: "We can't use System.Runtime.InteropServices.Marshal.IsComObject(obj) since it doesn't work in partial trust."
Partial trust is a concept of Code Access Security (CAS)/Security-Transparent. CAS and Security-Transparent are no longer supported in .NET Core, so we should use `Marshal.IsComObject` instead.
- Make switch-statement report correct error position when it fails to evaluate the condition.
- Make for-statement report correct error position when it fails to evaluate the initializer.
- For the condition of `if/for/while/do-while/do-until` statements, the sequence point update is either duplicate in some cases (for `if/for/while`) which causes debugger to stop at the condition twice before moving forward, or missing (for `do-while/do-until`) which causes debugger to skip the condition. They are fixed.
- `ConvertFrom-Markdown` is used for converting a markdown document or string to a MarkdownInfo object. It can optionally return an HTML or a VT100 encoded string in addition to an AST of the markdown document.
- `Show-Markdown` is used to either display the VT100 encoded string on the console or redirect the HTML string to the browser.
- `Set/Get-MarkdownOption` cmdlets are used to view or set markdown rendering options.
* The major refactoring changes are:
- In `ModuleIntrisic.cs`, remove unneeded Windows-PowerShell-only code.
- In `ModuleUtils.cs`
- use the new API `Directory.GetDirectories(string path, string searchPattern, EnumerationOptions enumerationOptions)` and `Directory.GetFiles(string path, string searchPattern, EnumerationOptions enumerationOptions)` to enumerate files and sub-directories within a directory path.
- remove the unused parameter `bool force` from `GetDefaultAvailableModuleFiles(bool force, bool isForAutoDiscovery, ExecutionContext context)`
- refactor the method `GetModuleVersionsFromAbsolutePath`. Add more comments and rename the method name.
- In `ModuleCmdletBase.cs`, refactor the method `GetModuleForNonRootedPaths` to `GetModuleForNames` to simply its implementation.
- In `PSModuleInfo.cs`
- group the declarations of `Declared*Exports` fields together
- rename `_detected*Exports` fields to `Detected*Exports` to group them together. They are internal fields and used outside `PSModuleInfo`.
* There is some perf improvement after this refactoring change:
- For `Get-Module -ListAvailable`, there is about 36% speed improvement for 94 default modules.
- For `Get-Module -ListAvailable -All`, there is about 14% speed improvement for totally 600 module files.
- For `Get-Module <name> -ListAvailable -List`, there is over 17x speed improvement for finding 13 modules from 600 modules. This is because we now filter names using the module file before creating a `PSMdouleInfo` object.
The fix causes a problem in debugging. When stepping over the scripts in debugging mode, the debugger will stop at the switch statement condition expression twice at the very beginning -- one for evaluating the condition, and one for iterating the condition.
After parsing `if () { }`, new lines are skipped in order to see if either `elseif` or `else` is present. When neither is present, we should resync back to the pointer before skipping those possibly available new lines, so that the new line tokens can be utilized by the subsequent parsing.
Refactor code to make it easier to maintain and a little faster. Changes are as follows:
1. Support finding a matching signature with variance. But make PowerShell prefer exact match over a match with variance.
2. The metadata signatures in `PSMethod<..>` are generated based on the array of method overloads in `MethodCacheEntry.MethodInformationStructures`, in the exact same order. So in `LanguagePrimitive.ConvertViaParseMethod`, when we try to figure out if there is a match using the metadata signatures in `PSMethod<..>`, we can get the index of the matching signature, and the same index should locate the matching method in `MethodCacheEntry.MethodInformationStructures`. Therefore, we don't need to compare signatures again in the actual conversion method, and instead, we can just leverage the index we found when figuring out the conversion in `ConvertViaParseMethod`.
- This gets rid of the reflection call `GetMethod("Invoke")` and the subsequent signature comparisons in the final conversion method.
- Also, when comparing signatures using `PSMethod<..>` in `ConvertViaParseMethod`, we can just use the generic argument types of each `Func<..>` metadata type, instead of calling `GetMethod("Invoke")` and then `GetParameters()`. This makes the code for comparing signatures simpler (the type `SignatureComparator`).
- Move `MatchesPSMethodProjectedType` from `PSMemberInfo.cs` to the type `SignatureComparator` in `LanguagePrimitives.cs`, as it's closely related to the signature comparison. Also, renamed it to `ProjectedTypeMatchesTargetType`.
- These changes make PSMethod-to-Delegate conversion a little faster, but no big improvement, as the true bottleneck probably is in delegate creation(?). Actually, the performance of this conversion is not critical at all at this moment because this feature should rarely be used in any hot script path. So this exercise is mainly for fun.
3. Remove `PSEnum<T>`. We can directly use enum types when constructing the metadata type `Func<..>`.
4. Remove the code that generates metadata signatures for generic method definitions (call `MakeGenericMethod` with fake types like `GenericType0`, `GenericType1`). This is because:
- We don't support convert generic method to delegate today, so may be better not spending time on preparing the metadata signature types for those methods.
- When the day comes that we need to support it, it's better to use generic argument types directly to construct the `Func<..>` metadata types. I left comments in `GetMethodGroupType` method in `PSMemberInfo.cs` to explain why that approach is better.
This regression was introduced by #6523, in `PSModuleInfo.cs`. A circular nested module check was removed because the comment there suggested it happens only with a deprecated workflow module. This causes a `StackOverflow` exception when running into circular nested modules.
Circular nested modules could happen for a module that is not well structured. For example, the module folder `test` contains two files: `test.psd1` and `test.psm1`, and `test.psd1` has the following content:
@{ ModuleVersion = '0.0.1'; RootModule = 'test'; NestedModules = @('test') }
The same value `test` is put in both RootModule and NestedModule, which will end up with a module whose nested module points to itself.
There are two changes in this PR:
1. Add back the check for circular nested modules in `PSModuleInfo.cs`.
2. Remove a wrong `Dbg.Assert` in `ModuleCmdletBase.cs` and two checks before it.
- For the assertion `Dbg.Assert(newManifestInfo.SessionState == ss`, when facing the example above, the nested module will first be loaded with a different session state, and then when trying to load the root module, the same loaded nested module will be reused for it. So 'newManifestInfo.SessionState' is not `ss`. The assertion will fail in that case.
- For the two checks before the assertion, they are not needed anymore based on the comments there.
Update 'Get-ChangeLog' to make it more accurate
Use "||" as the delimiter instead of "|", so it works for commit message that has the pipeline operator in it.
Handle merges that create a merge node in history.
Do not treat powershell team members as external contributors.
[breaking change] Clean up code related to the uses of `CommandTypes.Workflow` and `WorkflowInfo` in `System.Management.Automation`. This change mainly affects help provider code.
This PR includes a few minor breaking changes:
- Change the `public` constructors of `WorkflowInfo` to `internal`. We don't support workflow in PSCore, so I think it makes sense to not allow people to create `Workflow` instances.
- Remove the type `System.Management.Automation.DebugSource` since it's only used for workflow debugging.
- Remove the overload of `SetParent` from the abstract class `Debugger` that is only used for workflow debugging.
- Remove the same overload of `SetParent` from the derived class `RemotingJobDebugger`.
Replace `~` with `-` for preview download links in README.md
The file names have `~` in them, but when uploading files to Github, `~` will be automatically replaced. So we use `-` for the packages uploaded to Github.
Dongbo Wang